Unsupervised domain adaptive medical image segmentation method based on style consistency

Abstract

The invention discloses an unsupervised domain adaptive medical image segmentation method and system based on style consistency, and relates to the technical field of image processing, and the method comprises the steps: collecting CT abdominal organ scanning images comprising a plurality of source domains and target domains, and constructing an original training set; an SCUDA segmentation network is constructed, a local phase enhancement style fusion strategy is designed through a first sub-network to carry out image conversion and image reconstruction, and inconsistent mapping is found in a second sub-network through intermediate target domain synthesis images of different styles. The domain difference of the pancreatic region between the target domain synthetic image and the target domain real image is quantitatively analyzed, so that the integrity of the pancreatic region is further improved; and a style consistency entropy is defined on the target domain image so as to be more focused on an inconsistent region, thereby further optimizing the integrity of the pancreatic region. And the two sub-networks train updating parameters at the same time to obtain segmented images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an unsupervised domain adaptive medical image segmentation method and system based on style consistency. Background Art

[0002] Chronic pancreatitis (CP) is a persistent inflammation of the pancreas that can lead to permanent structural damage such as fibrosis and ductal stenosis of the pancreas, and is often accompanied by the formation of stones, which may eventually lead to pancreatic sclerosis, calcification foci, and even pancreatic cancer, seriously affecting the patient's quality of life and health. Computed tomography is currently the most important examination method for pancreatic diseases. Non-contrast-enhanced CT (NCECT) scan images help identify the structure and density of the pancreas, and high-density lesions in the pancreas (calcification and pancreatic duct stones) appear as high-density shadows on NCE images, which form a clear contrast with the surrounding tissues and are conducive to observation. Contrast-enhanced CT (CECT) scan images can better observe the blood vessels and other tissue structures around the pancreas. Therefore, the combination of NCECT and CECT dual-phase diagnosis of the pancreas and its high-density lesions can provide doctors with important anatomical and pathological information. Traditional pancreas segmentation algorithms usually rely on a large amount of labeled data for training. However, the pancreas of patients with chronic pancreatitis accounts for a very small proportion of CT image sequences, has strong individual morphological specificity, and is adhered to surrounding tissues and organs. In addition, NCECT and CECT images are not aligned and are uneven in style and texture. Therefore, the generalization effect of the segmentation network between NCECT and CECT images is poor. Since multiple phases of labeling tasks require a lot of time and effort from doctors, unsupervised domain adaptation (UDA) segmentation from a labeled domain (source domain) to another unlabeled domain (target domain) has become an urgent problem to be solved.

[0003] Much research in pattern recognition, image processing, and computer vision has been devoted to transferring images from one domain to another. These methods usually require a set of paired images that are identical in content and structure but different in style, assuming that there are certain latent relationships between paired images of different styles. By learning these relationships, the style transfer of images can be achieved. However, obtaining paired images is a challenging task and sometimes even impractical. Recently, inspired by generative adversarial networks (GANs) and cycle-consistent adversarial networks (CycleGANs), many studies have achieved unsupervised domain adaptation (UDA) in medical image analysis by aligning image appearance or latent features without the need for paired data. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is that the existing pancreas segmentation algorithm has the problem of requiring a large amount of data, having a poor generalization effect, and how to identify pancreas images with tissue adhesion or strong specificity.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: an unsupervised domain adaptive medical image segmentation method based on style consistency, comprising constructing a SCUDA network, performing image reconstruction through the first sub-network, and performing image segmentation through the second sub-network. and source domain image Input into the first sub-network to generate the target domain synthetic image , source domain reconstructed image , source domain synthetic image , target domain reconstructed image . The target domain image Corresponding content features , source domain image Corresponding content features Synthesize images with target domain Corresponding content features Input into the second sub-network to generate the segmentation prediction map , and In the first sub-network, a phase consistency discriminator is used to distinguish the phase consistency features corresponding to the source domain content features and the target domain content features, and to enhance the domain invariant content feature encoder. , style encoder of the source domain and the style encoder of the target domain The decoupling of the domain-invariant content feature encoder and the removal of domain-specific features from the domain-invariant content feature encoder are performed. The style inconsistency estimation is used in the second sub-network to perform segmentation predictions on the target domain synthetic images of different styles and generate inconsistency mapping areas, and the misclassified areas are measured to reduce the domain shift between the target domain synthetic images and the true target domain images. The style consistency entropy, segmentation consistency loss and segmentation loss are used in the second sub-network. The two sub-networks are trained and updated simultaneously to obtain the segmented images.

[0007] As a preferred solution of the unsupervised domain adaptive medical image segmentation method based on style consistency described in the present invention, wherein: the first sub-network includes a domain invariant content feature encoder , style encoder of the target domain , style encoder of the source domain , image reconstruction decoder , Local phase enhancement style fusion module LPSF, Phase enhancement style fusion module PSF, Discriminator for source domain synthetic images , discriminator for synthetic images in the target domain , for the phase consistency discriminator .

[0008] The second sub-network includes a multi-stage style consistency module MSSC and an image segmenter G,

[0009] It is the backbone network of ResNet34. and Adoption and The same architecture is used in G. The network has 5 convolutional layers. The first two convolutional layers are composed of residual blocks, and each of the last three layers is composed of transposed convolution, BN normalization layer and ReLU activation layer. It uses the same architecture as G.

[0010] The discriminator uses a network with four convolutional layers and one fully connected layer. The convolution in each convolutional layer is followed by an IN normalization layer and a LeakyReLU activation layer.

[0011] As a preferred solution of the unsupervised domain adaptive medical image segmentation method based on style consistency of the present invention, wherein: the image reconstruction by the first sub-network includes converting the source domain image into and the target domain image Send them to the domain invariant content feature encoder Get and , and simultaneously sent to and Extracting style features from the source domain and the style features of the target domain ,Will and The local phase enhancement content features of the source domain are generated by the LPSF module and the local phase enhanced style fusion feature of the target domain .

[0012] LPSF consists of a local content feature extraction submodule LCFE and a phase enhancement style fusion module PSF. , and the source domain label Combined input into LCFE, using from Extract the local vector , expressed as: ;

[0013] in, is the number of target vectors, is a reshape operation. yes The number of channels, then Input into a pooling layer, followed by three convolution operations to obtain three dynamic convolution kernel parameters , and , the three dynamic convolution kernel parameters are respectively Perform convolution operation and then unify the features to The same size and concatenated in the channel dimension to obtain the local content features of the source domain .

[0014] The LPSF module uses fast Fourier transform (FFT) to extract local content features in the source domain. The corresponding phase spectrum , the PSF module converts the phase spectrum and The features of the source domain image are added and enhanced, expressed as: ; in, is the inverse fast Fourier transform.

[0015] Style features of the target domain Take FFT operation and extract The amplitude spectrum of and phase spectrum , for the target domain amplitude spectrum Perform low-pass filtering to remove target domain noise, obtain pure target domain style, and enhance style fusion features in the local phase of the target domain It is expressed as: ; in, is the dot product operation, is the mask of the low-pass filter, It means that convolution, average pooling and full connection operations are performed between the two inputs first.

[0016] As a preferred solution of the unsupervised domain adaptive medical image segmentation method based on style consistency of the present invention, wherein: the image reconstruction through the first sub-network also includes generating a style-consistent image with the same style as the source domain image Same structure content but different from the target domain image Synthesized images in the target domain of the same style ,Will Input decoder , and at the same time Embedded in decoder In each layer of , a synthetic image of the target domain with adaptive instance normalization is generated , expressed as: ; in, represents the variance operation, represents the average operation, It is in the decoder The intermediate output features obtained by the upsampling operation performed in .

[0017] Style features of the source domain Take FFT operation and extract The amplitude spectrum of and phase spectrum , for the source domain amplitude spectrum Low-pass filtering is performed to remove source domain noise, obtain the pure source domain style, and enhance the style fusion feature of the local phase of the source domain It is expressed as: ; Will and Input into the LPSF module to generate the source domain reconstructed image , expressed as: ; in, It is the local phase enhanced style fusion feature of the source domain.

[0018] Will and Input into the PSF module to generate the target domain phase enhancement content feature Style fusion features enhanced by phase enhancement in source domain , expressed as: ; ; in, It is through The phase spectrum obtained by FFT is and Respectively through Perform FFT to obtain the amplitude and phase spectra.

[0019] Source domain synthetic image Through the decoder Use AdaIN to generate Input decoder , and at the same time Embedded in decoder In each layer of , expressed as: ; in, It is in the decoder The intermediate output features obtained by the upsampling operation performed in .

[0020] Will and Input into the PSF module to generate the target domain phase enhancement content feature and target domain phase enhanced style fusion features , expressed as: ; ; Target domain reconstructed image Through the decoder Use AdaIN to generate Input Decoder , and at the same time Embedded in decoder In each layer of , a target domain reconstructed image with adaptive instance normalization is generated , expressed as: ; in, It is the target domain phase enhanced style fusion feature.

[0021] As a preferred solution of the unsupervised domain adaptive medical image segmentation method based on style consistency described in the present invention, wherein: a phase consistency discriminator is constructed , distinguishing the phase consistency features corresponding to the source domain content features and the target domain content features, the loss function corresponding to the discriminator during training is as follows: ; in, Indicates expectation, introducing a discriminator for synthetic images in the target domain For adversarial learning, the loss function corresponding to the discriminator during training is expressed as: ; Generate source domain synthetic images and target domain reconstructed images without losing domain invariant information, and introduce a discriminator for source domain synthetic images For adversarial learning, the loss function corresponding to the discriminator during training is expressed as: ; The dual reconstruction consistency loss is constructed and expressed as: .

[0022] As a preferred solution of the unsupervised domain adaptive medical image segmentation method based on style consistency described in the present invention, wherein: the second sub-network obtains an inconsistency map from the content features corresponding to the target domain synthetic images of different styles and the content features corresponding to the target domain real images, measures the difficult area and constrains it, and assumes that by inputting the target domain synthetic images of different training periods into the encoder The high-level features obtained in ,in Indicates the epoch number of the training period. In the training of the style transfer network, select the The synthetic target domain image of epochs is expressed as ,in is the total number of selected epochs. The style consistency in the style transfer process is measured by the multi-stage style consistency module. The MSSC module consists of Split Head Composition, use Generate prediction probabilities of different styles for organs of interest , an inconsistency map is obtained by predicting probabilities of multiple different styles It is expressed as: ; in, yes The Class probability, yes The Class probability.

[0023] By exploiting style inconsistency, we define a multi-stage style consistency loss Focus on the head, tail and edge of the organ, as well as the adjacent organ tissue area, expressed as: ; in, yes Middle Class The predicted probability of pixels, is the source domain image The real label, yes Middle Class The true label of each pixel, yes Middle Class Inconsistency map of pixels.

[0024] As a preferred solution of the unsupervised domain adaptive medical image segmentation method based on style consistency described in the present invention, wherein: the two sub-networks are simultaneously trained to update parameters to obtain the segmented image, including the target domain image Generate predicted probabilities of organs of interest using segmentation heads of different styles from pre-trained models , and generate corresponding prediction results ,like If the foreground value of the same pixel in the prediction results is the same, it is considered a reliable prediction result. .

[0025] The target domain image Feed into the domain-invariant content feature encoder which is constantly updating , generated is fed into the target domain image segmentation head being updated , yielding the predicted probability of the organ of interest and prediction results , by comparing the two prediction results and Perform an XOR operation to obtain an inconsistent mapping of the target domain image , define the style consistency entropy loss, expressed as: ; in, Is an inconsistent mapping Middle Category pixels, is the predicted probability map of the organ of interest Middle Category pixels, The value of is set to 1e-7.

[0026] Using the obtained reliable consistency area The source domain images and target domain images are jointly trained, and the target domain images are transformed into Input the target segmentation head to generate two predicted probability maps of the organ of interest and , during the training process, the segmentation consistency loss of the output for the target domain image is expressed as: ; Training on unlabeled target domain images via segmentation loss , expressed as: ; in, yes Middle Class pixels, for The Class pixels, for The Class pixels, for The Class pixels.

[0027] Define a segmentation loss , expressed as: ; in, for The Class pixels, for The Class pixels.

[0028] The overall style consistency segmentation loss is expressed as: ;

[0029] By optimizing to segment the target organ.

[0030] Another object of the present invention is to provide an unsupervised domain adaptive medical image segmentation system based on style consistency, which can perform segmentation prediction on target domain synthetic images of different styles and generate inconsistency mapping areas through a style inconsistency estimation method, so as to measure the easily misclassified areas to reduce the domain offset between the target domain synthetic images and the real target domain images, and improve the integrity of the target organ, thereby solving the problem that the current traditional pancreatic segmentation algorithm contains NCECT and CECT images that are unable to be aligned and are uneven in style and texture.

[0031] As a preferred solution of the unsupervised domain adaptive medical image segmentation system based on style consistency described in the present invention, it includes: a data set construction module, a SCUDA segmentation network optimization module, and a parameter update module. The data set construction module is used to collect CT abdominal organ scan images containing multiple source domains and target domains to construct an original training set. The SCUDA segmentation network optimization module is used to construct a SCUDA segmentation network, reconstruct the image through the first sub-network, and generate an inconsistent mapping area through the prediction map of different segmented target domain images in the second sub-network. The parameter update module is used to simultaneously train and update parameters of the two sub-networks to obtain the segmented image.

[0032] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of an unsupervised domain adaptive medical image segmentation method based on style consistency.

[0033] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an unsupervised domain adaptive medical image segmentation method based on style consistency.

[0034] Beneficial effects of the present invention: The style inconsistency estimation method of the unsupervised domain adaptive medical image segmentation method based on style consistency provided by the present invention designs a local phase enhancement style fusion strategy for image conversion and image reconstruction through the first sub-network, finds inconsistent mappings in the intermediate target domain synthetic images of different styles in the second sub-network, and quantitatively analyzes the domain difference between the target domain synthetic image and the target domain real image of the pancreas region to further improve the integrity of the pancreas region. The prediction map of the target domain image with different segmentations is used to generate the inconsistent mapping area. And the style consistency entropy is defined on the target domain image to focus more on the inconsistent area, so as to further optimize the integrity of the pancreas region. The two sub-networks are trained and updated with parameters at the same time to obtain the segmented image. The style inconsistency estimation method of the unsupervised domain adaptive medical image segmentation method based on style consistency provided by the present invention performs segmentation prediction on the target domain synthetic images of different styles and generates the inconsistent mapping area, so as to measure the easily misclassified area to reduce the domain offset between the target domain synthetic image and the real target domain image, and improve the integrity of the target organ. The pancreas segmentation result of the present invention achieves better results in Recall, Precision, Jac and DSC indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0036] Figure 1 An overall flow chart of an unsupervised domain adaptive medical image segmentation method based on style consistency provided for the first embodiment of the present invention.

[0037] Figure 2 Another flowchart of an unsupervised domain adaptive medical image segmentation method based on style consistency provided for the first embodiment of the present invention.

[0038] Figure 3 A schematic diagram of a local phase enhancement style fusion (LPSF) module of an unsupervised domain adaptive medical image segmentation method based on style consistency provided in the first embodiment of the present invention.

[0039] Figure 4 A schematic diagram of a phase enhancement style fusion (PSF) module of an unsupervised domain adaptive medical image segmentation method based on style consistency provided in the first embodiment of the present invention.

[0040] Figure 5 The present invention provides a first embodiment of a target domain synthetic image at different training stages of an unsupervised domain adaptive medical image segmentation method based on style consistency.

[0041] Figure 6 An overall flow chart of an unsupervised domain adaptive medical image segmentation system based on style consistency is provided for the third embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0043] Example 1

[0044] Reference Figure 1-Figure 5 , as an embodiment of the present invention, provides an unsupervised domain adaptive medical image segmentation method based on style consistency, comprising:

[0045] S1: Collect multiple CT abdominal organ scan images containing source domains and target domains to construct the original training set.

[0046] Further, refer to Figure 1 As shown, it is a flowchart of the overall process of the present invention. In this embodiment, the data acquisition used by the present invention includes a total of 127 dual-phase 3D abdominal CT images of patients with chronic pancreatitis. The axial size of each non-contrast enhanced (NCE) and contrast enhanced (CE, venous phase) CT image is 512×512, and the dual-phase CT images of each patient are taken at different time points using different scanning methods. In order to maintain the consistency of image size in the two data sets, all images are uniformly adjusted to 256×256. Due to the different acquisition methods of NCECT and CECT images and individual differences in patients, the number of cross-sectional images ranges from [35, 120]. The sagittal and coronal plane intervals of NCECT images and CECT images are [0.51, 0.79] mm, and the axial plane interval is 1 mm. A total of 9865 NCECT and 9978 CECT two-dimensional (2D) axial images were obtained. Since it is difficult to annotate high-density lesions (stones and calcifications in the pancreas) in CECT images, only the annotation information of the pancreas is available in CECT images, while the pancreas and its high-density lesion areas are annotated in NCECT images. All pancreatic and high-density lesion areas are manually annotated by professional doctors, with the pixels in the pancreatic area annotated as 1, the pixels in the high-density lesion area annotated as 2, and other areas annotated as 0. In order to prove that the method proposed in this chapter is more fair and reliable, a 2-fold cross-validation scheme was adopted, with 65 and 62 patients in each group, respectively. The data of the i-th (i=1, 2) fold was used as the evaluation set, and the data of the remaining folds was used as the training set. Two models were trained. The evaluation indicators of the two models on their respective evaluation sets were averaged as the final result. In terms of data preprocessing, the original intensity values ​​of the CT images were first truncated to [-100, 240] or [900, 1300] to enhance the details of the pancreas and its high-density lesion areas. Secondly, all data images were normalized to reduce the data differences caused by the medical image acquisition process and obtain a relatively uniform data set. Finally, due to the diversity of the pancreas and its high-density lesion locations, data augmentation such as random flipping, random noise, and random rotation within [-30, 30] degrees was performed on the images before training to avoid model overfitting.

[0047] It should be noted that the present invention uses the pytorch deep learning framework, torch version 1.13.0. All training and verification processes are completed on an NVIDIA GeForce RTX 3090 graphics card equipped with 24G video memory. During the training process, the neural network uses a small batch method to read data, and the batch size is set to 2. The experiments all use the Adaptive Moment Estimation (Adam) optimization algorithm, where , , the initial learning rate is set to , the training cycle epochs is set to 120.

[0048] It should also be noted that the first subnetwork includes the domain-invariant content feature encoder , style encoder of the target domain , style encoder of source domain , image reconstruction decoder , Local phase enhancement style fusion module LPSF, Phase enhancement style fusion module PSF, Discriminator for source domain synthetic images , discriminator for synthetic images in the target domain , for the phase consistency discriminator .

[0049] The second sub-network includes a multi-stage style consistency module MSSC and an image segmenter G.

[0050] It is the backbone network of ResNet34. and Adoption and The same architecture is used in G. The network has 5 convolutional layers. The first two convolutional layers are composed of residual blocks, and each of the last three layers is composed of transposed convolution, BN normalization layer and ReLU activation layer. Uses the same architecture as G;

[0051] The discriminator uses a network with four convolutional layers and one fully connected layer. The convolution in each convolutional layer is followed by an IN normalization layer and a LeakyReLU activation layer.

[0052] S2: Build the SCUDA segmentation network, reconstruct the image through the first sub-network, and generate the inconsistency mapping area through the prediction map of different segmentation target domain images in the second sub-network.

[0053] Furthermore, we construct a SCUDA segmentation network, which includes three encoders (domain-invariant content feature encoder , the style encoder of the source domain and the style encoder of the target domain ), 1 style transfer decoder , 1 Local Phase enhanced Style Fusion (LPSF) module, 1 Phase enhanced Style Fusion (PSF) module, 2 discriminators for source and target domain synthetic images, phase consistency discriminator , pancreas segmentation head G.

[0054] It should be noted that the image reconstruction by the first sub-network includes converting the source domain image and the target domain image Send them to the domain invariant content feature encoder Get and , and simultaneously sent to and Extracting style features from the source domain and the style features of the target domain ,Will and The local phase enhancement content features of the source domain are generated by the LPSF module and the local phase enhanced style fusion feature of the target domain .

[0055] LPSF consists of a local content feature extraction submodule LCFE and a phase enhancement style fusion module PSF. and the source domain label Combined input into LCFE, using from Extract the local vector , expressed as: ; in, is the number of target vectors, is a reshape operation. yes The number of channels, then Input into a pooling layer, followed by three convolution operations to obtain three dynamic convolution kernel parameters and , the three dynamic convolution kernel parameters are respectively Perform convolution operation and then unify the features to The same size and concatenated in the channel dimension to obtain the local content features of the source domain .

[0056] The LPSF module uses fast Fourier transform (FFT) to extract local content features in the source domain. The corresponding phase spectrum , the PSF module converts the phase spectrum and The features of the source domain image are added and enhanced, expressed as: ; in, is the inverse fast Fourier transform.

[0057] Style features of the target domain Take FFT operation and extract The amplitude spectrum of and phase spectrum , for the target domain amplitude spectrum Perform low-pass filtering to remove target domain noise, obtain pure target domain style, and enhance style fusion features in the local phase of the target domain It is expressed as: ; in, is the dot product operation, is the mask of the low-pass filter, It means that convolution, average pooling and full connection operations are performed between the two inputs first.

[0058] It should also be noted that the image reconstruction by the first sub-network also includes generating an image with the same Same structure content but different from the target domain image Synthesized images in the target domain of the same style ,Will Input Decoder , and at the same time Embedded in decoder In each layer of , a synthetic image of the target domain with adaptive instance normalization is generated , expressed as: ; in, represents the variance operation, represents the average operation, It is in the decoder The intermediate output features obtained by the upsampling operation performed in .

[0059] Style features of the source domain Take FFT operation and extract The amplitude spectrum of and phase spectrum , for the source domain amplitude spectrum Low-pass filtering is performed to remove source domain noise, obtain the pure source domain style, and enhance the style fusion feature of the local phase of the source domain It is expressed as: ; Will and Input into the LPSF module to generate the source domain reconstructed image , expressed as: ; in, It is the local phase enhanced style fusion feature of the source domain.

[0060] Will and Input into the PSF module to generate the target domain phase enhancement content feature Style fusion features enhanced by phase enhancement in source domain , expressed as: ; in, It is through The phase spectrum obtained by FFT is and Respectively through Perform FFT to obtain the amplitude and phase spectra.

[0061] Source domain synthetic image Through the decoder Use AdaIN to generate Input Decoder , and at the same time Embedded in decoder In each layer of , expressed as: ; in, It is in the decoder The intermediate output features obtained by the upsampling operation performed in .

[0062] Will and Input into the PSF module to generate the target domain phase enhancement content feature and target domain phase enhanced style fusion features , expressed as: ;

[0063] Target domain reconstructed image Through the decoder Use AdaIN to generate Input decoder , and at the same time Embedded in decoder In each layer of , a target domain reconstructed image with adaptive instance normalization is generated , expressed as: ; in, It is the target domain phase enhanced style fusion feature.

[0064] Furthermore, we construct a phase consistency discriminator , distinguishing the phase consistency features corresponding to the source domain content features and the target domain content features, the loss function corresponding to the discriminator during training is as follows: ; in, Indicates expectation, introducing a discriminator for synthetic images in the target domain For adversarial learning, the loss function corresponding to the discriminator during training is expressed as: ; Generate source domain synthetic images and target domain reconstructed images without losing domain invariant information, and introduce a discriminator for source domain synthetic images For adversarial learning, the loss function corresponding to the discriminator during training is expressed as: ; The dual reconstruction consistency loss is constructed and expressed as: .

[0065] It should be noted that since the source domain image and the target image Due to the significant style differences between the target organs and the contrast differences between the target organs and the surrounding tissues, it is difficult for the style transfer network to quickly and accurately transfer the style of the target domain image to the target organ, which leads to the target domain synthetic images in different training cycles. and the target domain real image There are still style differences between them. Therefore, training only with synthetic images of the target domain generated in a specific period without adding additional constraints may miss some parts of the target organs. To solve this problem, an inconsistency estimation method is proposed to synthesize images from target domains with different styles. An inconsistency map is obtained from the target domain real image to measure the difficult areas, and then they are constrained to reduce the difference between the target domain synthetic image and the target domain real image and improve the integrity of the target organ.

[0066] Furthermore, in the second sub-network, the inconsistency map is obtained from the content features corresponding to the target domain synthetic images of different styles and the content features corresponding to the target domain real images, and the difficult areas are measured and constrained. The high-level features obtained in ,in Indicates the epoch number of the training period. In the training of the style transfer network, select the The synthetic target domain image of epochs is expressed as ,in is the total number of epochs selected. The style consistency in the style transfer process is measured by the multi-stage style consistency module. The MSSC module consists of Split Head Composition, use Generate prediction probabilities of different styles for organs of interest , an inconsistency map is obtained by predicting probabilities of multiple different styles It is expressed as: ; in, yes The Class probability, yes The Class probability.

[0067] By exploiting style inconsistency, we define a multi-stage style consistency loss Focus on the head, tail and edge of the organ, as well as the adjacent organ tissue area, expressed as: ; in, yes Middle Class The predicted probability of pixels, is the source domain image The real label, yes Middle Class The true label of each pixel, yes Middle Class Inconsistency map of pixels.

[0068] S3: The two sub-networks are trained and parameter updated simultaneously to obtain the segmented image.

[0069] Furthermore, due to the huge differences between different domains, there is always a certain gap between the intermediate synthetic target domain images and the real target domain images at different training periods. In some regions of the organ of interest in the source domain image, they can be easily converted to the target domain and are therefore regarded as consistent regions, while other regions are difficult to convert to the target domain and are regarded as inconsistent regions. Therefore, the target domain image can be divided into consistent regions and inconsistent regions to further improve the integrity of the organ of interest through the target segmentation head.

[0070] It should be noted that the two sub-networks are trained and updated simultaneously to obtain the segmented image including the target domain image. Generate predicted probabilities of organs of interest using segmentation heads of different styles from pre-trained models , and generate corresponding prediction results ,like If the foreground value of the same pixel in the prediction results is the same, it is considered a reliable prediction result. .

[0071] The target domain image Feed into the domain-invariant content feature encoder which is constantly updating , generated is fed into the target domain image segmentation head being updated , yielding the predicted probability of the organ of interest and prediction results , by comparing the two prediction results and Perform an XOR operation to obtain an inconsistent mapping of the target domain image , define the style consistency entropy loss, expressed as: ; in, Is an inconsistent mapping Middle Category pixels, is the predicted probability map of the organ of interest Middle Category pixels, The value of is set to 1e-7.

[0072] Using the obtained reliable consistency area The source domain images and target domain images are jointly trained, and the target domain images are transformed into Input the target segmentation head to generate two predicted probability maps of the organ of interest and , during the training process, the segmentation consistency loss of the output for the target domain image is expressed as: ;

[0073] Training on unlabeled target domain images via segmentation loss , expressed as: ; in, yes Middle Class pixels, for The Class pixels, for The Class pixels, for The Class pixels.

[0074] Define a segmentation loss , expressed as: ; in, for The Class pixels, for The Class pixels.

[0075] The overall style consistency segmentation loss is expressed as: ; By optimizing to segment the target organ.

[0076] Example 2

[0077] An embodiment of the present invention provides an unsupervised domain adaptive medical image segmentation method based on style consistency. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0078] First, in order to quantitatively evaluate the performance of the method proposed in the present invention, the present invention uses four evaluation indicators commonly used in the field of image segmentation, namely, DSC coefficient (Dice-similarity coefficient, DSC), Jac coefficient (Jaccard similarity coefficient), precision (Precision) and recall rate (Recall) to measure the segmentation performance of each algorithm for pancreatic images. The calculation method of each indicator is as follows: ; TP (True Positive) represents the number of correctly predicted positive pixels, FP (False Positive) represents the number of incorrectly predicted positive pixels, and FN (False Negative) represents the number of negative pixels that were not correctly predicted. DSC is an important evaluation indicator in image segmentation tasks, which is used to measure the similarity between the predicted results and the standard results. Its value close to 1 indicates a high similarity. The Jac indicator represents the ratio of the intersection and union between the predicted results and the standard results, while the precision measures the proportion of pixels predicted correctly among the pixels predicted by the network as positive, and the recall rate refers to the proportion of positive pixels in the standard that are correctly predicted by the network.

[0079] In order to verify the effectiveness of the proposed unsupervised domain adaptive medical image segmentation algorithm based on style consistency, ablation experiments were conducted on the local phase enhancement style fusion method (LPSF), phase consistency (PC) method, multi-stage style inconsistency estimation (MSSC) method and style consistency entropy (SCE) method, and the framework that does not contain LPSF, PC, MSSC and SCE methods was used as the Baseline. Net3-1 is the result obtained by adding the LPSF method to the Baseline, Net3-2 adds the phase consistency method to Net3-1, Net3-3 adds the multi-stage style inconsistency estimation loss to Net3-2, and Net3-4 introduces the style consistency entropy loss at the end.

[0080] The NCECT image is used as the source domain, and the CECT image is used as the target domain. After the ablation experiment on the method of the present invention, Table 1 is obtained, in which the LPSF method of Net3-1 aims to alleviate domain shift and produce locally enhanced target organs, which is improved by 2.58%, 2.12%, and 1.88% respectively compared with the Baseline in Recall, Jac, and DSC. Net3-2 adds the PC method to further distinguish the phase consistency between the source domain content features and the target domain content features, enhance the decoupling between the domain-invariant content feature encoder and the style encoder, and remove the domain-specific features in the domain-invariant content feature encoder, which is improved by 4.18%, 1.04%, 3.79%, and 3.36% respectively compared with the Baseline in Recall, Precision, Jac, and DSC. In order to reduce the difference between the target domain synthetic images and the target domain real images and improve the integrity of the target organs, the MSSC method was added to Net3-3, which improved Recall, Precision, Jac, and DSC by 2.42%, 5.06%, 5.25%, and 4.44% respectively compared with the Baseline. Finally, by encouraging the network to pay more attention to areas with high uncertainty to produce better segmentation results, Net3-4 added the SCE method, which improved Recall, Precision, Jac, and DSC by 2.74%, 5.72%, 6.21%, and 5.2% respectively compared with the Baseline.

[0081] Table 1 Ablation experiment data table

[0082] The present invention compares the segmentation results with the traditional unsupervised domain adaptive segmentation learning network, as shown in Table 2. Table 2 compares the segmentation results of the present invention with others' methods.

[0083] Table 2 Multi-method data comparison table

[0084]

[0085] According to the above table analysis, DDANet, DSAN and DDFseg align them at the feature level and the predicted image level from the perspective of feature decoupling. However, in domain adaptive segmentation tasks with severe modality differences, such as small organ segmentation, they often lead to local deformation and a large number of incorrect segmentation results. In contrast, DLaST combines feature dissociation learning with self-training and introduces pixel-level shape constraints to alleviate the local deformation problem and improve domain adaptive performance. The DSC score of DLaST from NCECT to CECT is 65.62%, while the DSC score from CECT to NCECT is only 65.99%. In addition, MSYN and SABE perform low-frequency Fourier space exchange operations at the synthetic image level and the predicted image level, respectively, but due to the high data requirements, the domain adaptive segmentation performance is poor. On the basis of low-frequency exchange, MeFDA and DOCR also introduce adversarial learning and entropy minimization to enhance the performance of domain invariant feature extractors. Specifically, the DSC score of DOCR from NCECT to CECT is 68.61%, while that from CECT to NCECT is only 68.24%. SASAN focuses on solving the domain adaptation problem by using attention during the image transformation process. In addition, PNP, SIFA, and DSAN perform feature and image alignment, but all three methods achieve poor results in the small target pancreas segmentation task in this paper, and also produce some serious organ deformation during the image transformation process. In contrast, CycleGAN adopts image-level adversarial learning for cross-domain segmentation and obtains a DSC score of 70.47% in the case of NCECT to CECT, while it is 68.63% in the case of CECT to NCECT. However, Synseg integrates the CycleGAN method into an end-to-end framework, but does not achieve better results. Finally, UESM achieves good results due to its uncertainty-aware self-training strategy and feature recalibration module. The DSC score of UESM from NCECT to CECT is 74.72%, while that from CECT to NCECT is 66.44%. The SCUDA segmentation method proposed in this paper is different from most of the above UDA methods. This method focuses on the local area of ​​the target segmented organ, designs phase consistency and multi-stage style inconsistency to alleviate domain shift, and introduces style consistency entropy to effectively utilize unlabeled target domain data. Therefore, in terms of various indicators of cross-domain tasks in different phases of CT, this method has shown excellent performance and is superior to other UDA methods.

[0086] The CECT image is used as the source domain, and the NCECT image is used as the target domain. After the ablation experiment on the method of the present invention, Table 3 is obtained, in which the LPSF method of Net3-1 aims to alleviate domain shift and produce locally enhanced target organs, which is improved by 5.44%, 3.13%, and 2.59% respectively compared with the Baseline in Precision, Jac, and DSC. Net3-2 adds the PC method to further distinguish the phase consistency between the source domain content features and the target domain content features, enhance the decoupling between the domain-invariant content feature encoder and the style encoder, and remove the domain-specific features in the domain-invariant content feature encoder, which is improved by 1.74%, 1.04%, 3.76%, and 3.63% respectively compared with the Baseline in Recall, Precision, Jac, and DSC. In order to reduce the difference between the target domain synthetic images and the target domain real images and improve the integrity of the target organs, the MSSC method is added to Net3-3, which improves Recall, Precision, Jac, and DSC by 2.19%, 6.61%, 6.19%, and 5.32% respectively compared with the Baseline. Finally, by encouraging the network to pay more attention to areas with high uncertainty to produce better segmentation results, Net3-4 adds the SCE method, which improves Recall, Precision, Jac, and DSC by 3.5%, 7.34%, 7.32%, and 6.15% respectively compared with the Baseline. Table 3 Ablation experiment data table

[0087] Table 4 shows the results of the comparative experiments using CECT images as the source domain and NCECT images as the target domain. The best segmentation performance is bolded. As can be seen from the table, the Recall, Precision, Jac and DSC scores of the fully supervised model in the target domain are 76.39%, 78.66%, 64.29% and 75.53% respectively. When the segmentation model of the source domain is directly used to test the target domain, the Recall, Precision, Jac and DSC scores are 29.98%, 45.38%, 25.21% and 32.39% respectively, which indicates that there is a serious domain shift between NCECT and CECT images, which leads to a sharp decline in the performance of deep neural networks on the UDA image segmentation task. The SCUDA method achieved 78.37%, 75.27%, 62.82% and 74.57% in Recall, Precision, Jac and DSC respectively. In particular, the SCUDA method has made great progress in DSC, the most important indicator in the segmentation task, and also obtained the best segmentation effect.

[0088] Table 4 Multi-method data comparison table

[0089] Example 3

[0090] like Figure 6 An embodiment of the present invention is shown, which provides an unsupervised domain adaptive medical image segmentation system based on style consistency, including a data set construction module, a SCUDA segmentation network optimization module, and a parameter updating module;

[0091] Among them, the dataset construction module is used to collect multiple CT abdominal organ scan images containing source domains and target domains to construct the original training set; the SCUDA segmentation network optimization module is used to construct the SCUDA segmentation network, reconstruct the image through the first sub-network, and generate the inconsistent mapping area through the prediction map of different segmentation target domain images in the second sub-network; the parameter update module is used to simultaneously train and update the parameters of the two sub-networks to obtain the segmented image.

[0092] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0093] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0094] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk case (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0095] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logical function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0096] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An unsupervised domain adaptive medical image segmentation method based on style consistency, characterized in that: include: Construct the SCUDA network, perform image reconstruction through the first sub-network, and perform image segmentation through the second sub-network; The target domain image and source domain images Input into the first sub-network to generate the target domain synthetic image , source domain reconstructed image , source domain synthetic image , target domain reconstructed image ; The target domain image Corresponding content features , source domain image Corresponding content features Synthesize images with target domain Corresponding content features Input into the second sub-network to generate the segmentation prediction map , and ; Use the phase consistency discriminator in the first sub-network to distinguish the phase consistency features corresponding to the source domain content features and the target domain content features, and enhance the domain invariant content feature encoder , style encoder of the source domain and the style encoder of the target domain decoupling and removing domain-specific features from the domain-invariant content feature encoder; In the second sub-network, style inconsistency estimation is used to perform segmentation prediction on target domain synthetic images of different styles and generate inconsistency mapping areas, measuring the easily misclassified areas to reduce the domain shift between the target domain synthetic images and the real target domain images; Use style consistency entropy, segmentation consistency loss, and segmentation loss in the second sub-network; The two sub-networks are trained simultaneously to update parameters to obtain the segmented image.

2. The unsupervised domain adaptive medical image segmentation method based on style consistency according to claim 1, characterized in that: The first sub-network includes a domain-invariant content feature encoder , style encoder of the target domain , style encoder of the source domain , image reconstruction decoder , Local phase enhancement style fusion module LPSF, Phase enhancement style fusion module PSF, Discriminator for source domain synthetic images , discriminator for synthetic images in the target domain , for the phase consistency discriminator ; The second sub-network includes a multi-stage style consistency module MSSC and an image segmenter G, It is the backbone network of ResNet34. and Adoption and The same architecture is used in G. The network has 5 convolutional layers. The first two convolutional layers are composed of residual blocks, and each of the last three layers is composed of transposed convolution, BN normalization layer and ReLU activation layer. Uses the same architecture as G; The discriminator uses a network with four convolutional layers and one fully connected layer. The convolution in each convolutional layer is followed by an IN normalization layer and a LeakyReLU activation layer.

3. The unsupervised domain adaptive medical image segmentation method based on style consistency according to claim 2, characterized in that: The image reconstruction by the first sub-network includes converting the source domain image into and the target domain image Send them to the domain invariant content feature encoder Get and , and simultaneously sent to and Extracting style features from the source domain and the style features of the target domain ,Will and The local phase enhancement content features of the source domain are generated by the LPSF module and the local phase enhanced style fusion feature of the target domain ; LPSF consists of a local content feature extraction submodule LCFE and a phase enhancement style fusion module PSF. , and the source domain label Combined input into LCFE, using from Extract the local vector , expressed as: ； in, is the number of target vectors, is a reshape operation. yes The number of channels, then Input into a pooling layer, followed by three convolution operations to obtain three dynamic convolution kernel parameters , and , the three dynamic convolution kernel parameters are respectively Perform convolution operation and then unify the features to The same size and concatenated in the channel dimension to obtain the local content features of the source domain ; The LPSF module uses fast Fourier transform (FFT) to extract local content features in the source domain. The corresponding phase spectrum , the PSF module converts the phase spectrum and The features of the source domain image are added and enhanced, expressed as: ； in, is the inverse fast Fourier transform; Style features of the target domain Take FFT operation and extract The amplitude spectrum of and phase spectrum , for the target domain amplitude spectrum Perform low-pass filtering to remove target domain noise, obtain pure target domain style, and enhance style fusion features in the local phase of the target domain It is expressed as: ； in, is the dot product operation, is the mask of the low-pass filter, It means that convolution, average pooling and full connection operations are performed between the two inputs first.

4. The unsupervised domain adaptive medical image segmentation method based on style consistency according to claim 3, characterized in that: The image reconstruction by the first sub-network also includes generating an image having the same Same structure content but different from the target domain image Synthesized images in the target domain of the same style ,Will Input decoder , and at the same time Embedded in decoder In each layer of , a synthetic image of the target domain with adaptive instance normalization is generated , expressed as: ； in, represents the variance operation, represents the average operation, It is in the decoder The intermediate output features obtained by the upsampling operation performed in ; Style features of the source domain Take FFT operation and extract The amplitude spectrum of and phase spectrum , for the source domain amplitude spectrum Low-pass filtering is performed to remove source domain noise, obtain the pure source domain style, and enhance the style fusion feature of the local phase of the source domain It is expressed as: ； Will and Input into the LPSF module to generate the source domain reconstructed image , expressed as: ； in, It is the local phase enhanced style fusion feature of the source domain; Will and Input into the PSF module to generate the target domain phase enhancement content feature and source domain phase enhanced style fusion features , expressed as: ； ； in, It is through The phase spectrum obtained by FFT is and Respectively through Perform FFT to obtain the amplitude and phase spectra; Source domain synthetic image Through the decoder Use AdaIN to generate Input decoder , and at the same time Embedded in decoder In each layer of , expressed as: ； in, It is in the decoder The intermediate output features obtained by the upsampling operation performed in ; Will and Input into the PSF module to generate the target domain phase enhancement content feature and target domain phase enhanced style fusion features , expressed as: ； ； Target domain reconstructed image Through the decoder Use AdaIN to generate Input decoder , and at the same time Embedded in decoder In each layer of , a target domain reconstructed image with adaptive instance normalization is generated , expressed as: ； in, It is the target domain phase enhanced style fusion feature.

5. The unsupervised domain adaptive medical image segmentation method based on style consistency according to claim 4, characterized in that: Constructing a phase consistency discriminator , distinguishing the phase consistency features corresponding to the source domain content features and the target domain content features, the loss function corresponding to the discriminator during training is as follows: ； in, Indicates expectation, introducing a discriminator for synthetic images in the target domain For adversarial learning, the loss function corresponding to the discriminator during training is expressed as: ； Generate source domain synthetic images and target domain reconstructed images without losing domain invariant information, and introduce a discriminator for source domain synthetic images For adversarial learning, the loss function corresponding to the discriminator during training is expressed as: ； The dual reconstruction consistency loss is constructed and expressed as: 。 6. The unsupervised domain adaptive medical image segmentation method based on style consistency according to claim 5, characterized in that: The second sub-network obtains the inconsistency map from the content features corresponding to the target domain synthetic images of different styles and the content features corresponding to the target domain real images, measures the difficult area and constrains it. It is assumed that by inputting the target domain synthetic images of different training periods into the encoder The high-level features obtained in ,in Indicates the epoch number of the training period. In the training of the style transfer network, select the The synthetic target domain image of epochs is expressed as ,in is the total number of epochs selected. The style consistency in the style transfer process is measured by the multi-stage style consistency module. The MSSC module consists of Split Head Composition, use Generate prediction probabilities of different styles for organs of interest , an inconsistency map is obtained by predicting probabilities of multiple different styles It is expressed as: ； in, yes The Class probability, yes The Class probability; By exploiting style inconsistency, we define a multi-stage style consistency loss Focus on the head, tail and edge of the organ, as well as the adjacent organ tissue area, expressed as: ； in, yes Middle Class The predicted probability of pixels, is the source domain image The real label, yes Middle Class The true label of each pixel, yes Middle Class Inconsistency map of pixels.

7. The unsupervised domain adaptive medical image segmentation method based on style consistency according to claim 6, characterized in that: The two sub-networks are simultaneously trained to update parameters to obtain segmented images including the target domain image Generate predicted probabilities of organs of interest using segmentation heads of different styles from pre-trained models , and generate corresponding prediction results ,like If the foreground value of the same pixel in the prediction results is the same, it is considered a reliable prediction result. ; The target domain image Feed into the domain-invariant content feature encoder which is constantly updating , generated is fed into the target domain image segmentation head being updated , yielding the predicted probability of the organ of interest and prediction results , by comparing the two prediction results and Perform an XOR operation to obtain an inconsistent mapping of the target domain image , define the style consistency entropy loss, expressed as: ； in, Is an inconsistent mapping Middle Category pixels, is the predicted probability map of the organ of interest Middle Category pixels, The value of is set to 1e-7; Using the obtained reliable consistency area The source domain images and target domain images are jointly trained, and the target domain images are transformed into Input the target segmentation head to generate two predicted probability maps of the organ of interest and , during the training process, the segmentation consistency loss of the output for the target domain image is expressed as: ； Training on unlabeled target domain images via segmentation loss , expressed as: ； in, yes Middle Class pixels, for The Class pixels, for The Class pixels, for The Class pixels; Define a segmentation loss , expressed as: ； in, for The Class pixels, for The Class pixels; The overall style consistency segmentation loss is expressed as: ； By optimizing to segment the target organ.

8. A system using the unsupervised domain adaptive medical image segmentation method based on style consistency as claimed in any one of claims 1 to 7, characterized in that: Including dataset construction module, SCUDA segmentation network optimization module, parameter update module; The data set construction module is used to collect a plurality of CT abdominal organ scanning images including a source domain and a target domain to construct an original training set; The SCUDA segmentation network optimization module is used to construct a SCUDA segmentation network, perform image reconstruction through a first sub-network, and generate an inconsistency mapping region through prediction maps of different segmentation target domain images in a second sub-network; The parameter updating module is used to simultaneously train and update parameters of the two sub-networks to obtain the segmented image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the unsupervised domain adaptive medical image segmentation method based on style consistency described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the unsupervised domain adaptive medical image segmentation method based on style consistency described in any one of claims 1 to 7 are implemented.

Images