An unsupervised cerebral hemorrhage segmentation method based on multi-faceted domain adaptation technology

Abstract

An unsupervised cerebral hemorrhage segmentation method based on multi-level domain adaptation technology relates to the field of computer processing technology. The present invention uses an image synthesis network based on a semantic preservation mechanism, through multiple semantic preservation mechanisms, including semantic preservation of synthetic target domain images, semantic preservation of cyclically generated source domain images, and an adaptation mechanism for input images, to retain the semantic features of key areas for specific segmentation tasks; an image segmentation network based on a domain adaptation mechanism is used to optimize the semantic segmentation effect of CT cerebral hemorrhage segmentation tasks.

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and in particular to an unsupervised cerebral hemorrhage segmentation method based on multi-level domain adaptive technology. Background Art

[0002] Image semantic segmentation is an important research topic in the field of computer vision. Its goal is to segment an image into regions with different semantic information and annotate each region with a corresponding semantic label. For example, by performing image semantic segmentation on a street scene image, semantic labels can be added to objects in the image (such as roads, cars, pedestrians, trees, traffic lights, etc.).

[0003] With the development of artificial intelligence recognition technology, convolutional neural networks (CNNs) based on deep learning have attracted widespread attention and have been applied to the field of image semantic segmentation. Image semantic segmentation networks represented by FCN have achieved wide results in various fields.

[0004] Intracerebral hemorrhage refers to bleeding caused by non-traumatic rupture of blood vessels in the brain parenchyma, accounting for 20% to 30% of all strokes, with an acute mortality rate of 30% to 40%. Rapid determination of the extent of bleeding, early medical management and neurosurgery are crucial to promoting patient recovery.

[0005] Doctors usually make judgments based on cranial CT images to check the extent of cerebral hemorrhage. Considering the objectivity, repeatability, accuracy and urgency of the diagnostic results of cerebral hemorrhage, there is an urgent clinical requirement for automatic detection and quantitative analysis of cerebral hemorrhage areas. Therefore, taking effective methods to automatically extract bleeding areas can help to quickly and quantitatively judge the severity of bleeding, which has very important application value in clinical practice. However, obtaining a segmentation model with good generalization performance usually needs to be completed under supervised learning, which requires large-scale and professional data annotation. Therefore, it is necessary to propose a method for extracting cerebral hemorrhage areas from CT images under unsupervised learning.

[0006] The existing unsupervised segmentation methods based on synthetic virtual image technology first synthesize the target domain image, and then use the labels of existing data to train the segmentation model through unsupervised learning or semi-supervised learning domain adaptation methods. The basic idea is to adapt to the prediction of different fields by narrowing the marginal distribution of data. There are two methods for synthesizing target domain images: those based on manually defined metrics and those based on generative adversarial networks. However, the methods for synthesizing target domain images have the following shortcomings: for the synthesis method based on artificial metrics, the synthesized target domain image is highly dependent on the features extracted by the pre-trained network and the manually defined style metrics; the method of retaining the semantic information of the source domain is based on the measurement of the hidden layer representation, which is an indirect method and does not make targeted improvements to the semantic segmentation problem. For the synthesis method based on generative adversarial networks, such as directly applying cycleGAN, the synthesized target domain image usually loses the necessary semantic information in the segmentation task.

[0007] Considering the semantic similarity between MRI (magnetic resonance imaging) brain tumor images and CT brain hemorrhage images, the present invention uses the existing MRI brain tumor segmentation public data as source domain data. Summary of the invention

[0008] The present invention proposes a brain hemorrhage segmentation method based on multi-level domain adaptation technology. An image synthesis network based on a semantic preservation mechanism is adopted, and multiple semantic preservation mechanisms are used, including semantic preservation of synthetic target domain images, semantic preservation of cyclically generated source domain images, and adaptation mechanism of input images, to retain the semantic features of key areas for specific segmentation tasks; an image segmentation network based on a domain adaptation mechanism is adopted to optimize the semantic segmentation effect of CT brain hemorrhage segmentation tasks.

[0009] To achieve the above object, the present invention adopts the following technical solution:

[0010] A brain hemorrhage segmentation method based on multi-level domain adaptation technology is used to construct and train a brain CT image synthesis network based on a semantic preservation mechanism and a brain hemorrhage segmentation network based on a domain adaptation mechanism.

[0011] The above method comprises the following steps:

[0012] S1: Constructing a brain CT image synthesis network based on semantic preservation mechanism and a brain hemorrhage segmentation network based on domain adaptation mechanism

[0013] S11: The brain CT synthesis network based on the semantic preservation mechanism includes two generators, two discriminators and a semantic preservation module. Furthermore, the semantic preservation module contains two semantic segmentation sub-networks and a similar image selection module.

[0014] S12: Domain adaptive intracerebral hemorrhage segmentation network includes segmentation network and domain discriminator

[0015] S2: Train a CT image synthesis network based on a semantic preservation mechanism; further, labeled MRI brain tumor training data and unlabeled CT brain hemorrhage training data are required.

[0016] S3: Using the CT image synthesis network trained in step S2 and the labeled MRI image data, generate training data for subsequent training of the brain hemorrhage segmentation network based on the domain adaptation mechanism

[0017] S4: Train the CT brain hemorrhage segmentation network using the domain adaptation mechanism-based brain hemorrhage segmentation network defined in step S1 and the training data and corresponding MRI brain tumor labels generated in step S3 BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention is further described with reference to the accompanying drawings and embodiments.

[0019] Figure 1 This is an example of using cycleGAN to synthesize an image and compare it to the original image.

[0020] Figure 2 This is an example of comparing an image synthesized using a brain CT image synthesis network based on a semantic preservation mechanism to the original image.

[0021] Figure 3 is an example of an image matched by the similar image selection module (MRI image on the right and matched CT image on the left)

[0022] Figure 4 It is a schematic diagram of the process of the present invention

[0023] Figure 5 is a schematic diagram of an exemplary target domain data synthesis model provided by the present invention;

[0024] Figure 6 is a schematic diagram of an exemplary semantic segmentation model based on a domain adaptation mechanism provided by the present invention; DETAILED DESCRIPTION

[0025] Now the present invention is further described in detail with reference to the accompanying drawings, comprising the following steps:

[0026] S1: Constructing a brain CT image synthesis network based on semantic preservation mechanism and a brain hemorrhage segmentation network based on domain adaptation mechanism

[0027] S11: The brain CT image synthesis network based on the semantic preservation mechanism includes two structurally consistent generators, two structurally consistent discriminators and a semantic preservation module. Furthermore, the semantic preservation module contains two semantic segmentation sub-networks and an image semantically similar image selection module.

[0028] Furthermore, the generator includes a feature extraction stage, a feature fusion stage, and an upsampling stage.

[0029] The feature extraction stage is used to extract and encode features of the preprocessed input image. This stage includes 4 hidden layers: for the first two hidden layers, each hidden layer includes a 7x7 convolution operation, a batch normalization operation, and a rectified linear unit (ReLU) activation operation; for the last two hidden layers, each hidden layer includes a 3x3 convolution operation, a batch normalization operation, and a linear rectifier function activation operation.

[0030] Furthermore, the feature fusion stage re-encodes the feature map generated in the feature extraction stage to integrate the specific semantic information and abstract semantic information under different receptive fields. It includes a total of 6 densely connected modules (DenseNet block). The input of the hidden layer in the module adopts dense connection. Each densely connected block includes 5 hidden layers. Each hidden layer contains one step of 3x3 convolution operation, one step of layer normalization operation, and one step of linear rectification function activation operation.

[0031] Dense connections are defined as:

[0032] x l+1 =H([x0,...,x l ]) (1)

[0033] H(x i )=W*σ(B(x i )) (2)

[0034] d l =d0+g·l (3)

[0035] Among them, H(x i ) represents the input feature map x i Perform hidden layer operations, [] refers to the cascade operation of feature maps by channel, x i is the feature map of the i-th hidden layer input, l is the ordinal number of the hidden layer, d lrepresents the number of channels of the input of the lth hidden layer, g represents the number of channels of the output of all convolutional layers in the densely connected module, d0 represents the number of channels of the feature map of the first hidden layer of the input densely connected module, W represents the convolutional layer weights learned through training, B() represents the layer normalization operation, and σ() represents the linear rectification function activation operation.

[0036] The upsampling stage completes the conversion from feature encoding to the final output synthetic image, including two hidden layers, each of which contains a 3x3 transposed convolution operation, a layer normalization operation, and a tanh function activation operation.

[0037] Furthermore, the discriminator includes a feature extraction module and a prediction module, wherein the feature extraction module includes 5 hidden layers, each hidden layer includes a one-step convolution operation, a one-step layer normalization operation, and a one-step linear rectification function activation operation, and the prediction module is a one-step convolution operation, which is used to convert the feature map obtained by the feature extraction module into a prediction map.

[0038] The segmentation subnetwork of the semantic preservation module includes a pre-trained feature extraction module and multi-layer upsampling skip-layer connections. The pre-trained feature extraction module is a part of the VGG-16 network pre-trained on the ImageNet dataset, with a total of 5 convolution stages and 5 maximum pooling operations, wherein the first and second convolution stages contain two hidden layers, and the last three convolution stages each contain three hidden layers, each hidden layer contains a 3x3 convolution operation and a linear rectifier function activation operation.

[0039] The two sub-network model architectures are defined in the same way, both of which are pre-trained feature extraction modules combined with multi-layer upsampling skip-layer connection operations. The multi-layer upsampling skip-layer connection is defined as follows:

[0040] The feature maps output by the 5-step max pooling operation of the pre-trained feature extraction module are defined as v_out1, v_out2, v_out3, v_out4, and v_out5 in order;

[0041] Among them, v_out5 is subjected to deconvolution operation, linear rectification function activation operation, layer normalization operation, and then added to the feature map in v_out4 according to the element value to obtain the feature map temp1;

[0042] After deconvolution, linear rectification function activation, and layer normalization, temp1 is added to the feature map in v_out3 by element value to obtain feature map temp2.

[0043] After deconvolution, linear rectification function activation, and layer normalization, temp2 is added to the feature map in v_out2 according to the element value to obtain the feature map temp3;

[0044] After deconvolution, linear rectification function activation, and layer normalization, temp3 is added to the feature map in v_out1 according to the element value to obtain the feature map temp4;

[0045] After deconvolution, linear rectification function activation, and layer normalization, temp4 obtains the final feature map. After a 1x1 convolution and a sigmoid activation, the final feature map is thresholded to obtain the predicted segmentation result.

[0046] The similar image selection module in the semantic preservation module is responsible for finding the most similar CT image for each MRI image input. In order to avoid repeatedly selecting the same similar image and causing the model to only learn fixed mappings, it randomly selects from the 20 CT images with the highest similarity. The similarity is defined as follows:

[0047] sim=100d hanming (phash(x),phash(y))+|focus(x)-focus(y)| (4)

[0048] Here, x and y refer to the two images used to calculate the similarity, and phash() is an image-aware hashing algorithm that reduces the frequency of the image through discrete cosine transform (DCT) to obtain the binary fingerprint of the image. hanming () is the Hamming distance, which is used to obtain the similarity between binary fingerprints of images. focus() calculates the proportion of rough lesion areas in the image. For the source domain image, i.e., MRI image, the proportion of pixels with a value greater than 0 in the semantic segmentation label to the total pixels of the image can be directly taken; for the target domain image, i.e., CT image, the image is subjected to a threshold binarization operation: for pixel values ​​greater than 120 and less than 180, the value is set to 1, and the remaining pixel values ​​are set to 0.

[0049] S12: The image semantic segmentation network based on the domain adaptation mechanism includes a segmentation network and a domain discriminator.

[0050] Among them, the segmentation network structure is consistent with the segmentation sub-network structure in the semantic preservation module of the above-mentioned image synthesis network.

[0051] The domain discriminator input is the feature map output by the hidden layer and the prediction obtained by the hidden layer after the upsampling process. It includes 4 hidden layers and one classification layer. Each hidden layer includes a 4x4 convolution operation and a leaky linear rectifier function (Leaky ReLU) operation. The classification layer is a convolution layer used to output a prediction map with a dimension of 1.

[0052] S2: Training the target domain training data synthesis network

[0053] Furthermore, labeled source domain training data, i.e., MRI images and their semantic segmentation labels, and unlabeled target domain training data, i.e., brain CT images, are required. Since unpaired matching is used, the number of source domain and target domain images is not required to be consistent.

[0054] Preprocess and enhance the training data:

[0055] Among them, the preprocessing includes sampling the source domain image, source domain label, and target domain image to a consistent size, and normalizing all images, that is, processing the data to zero mean and unit variance;

[0056] Data augmentation method: including randomly floating the brightness values ​​of all images by 20% and horizontally flipping the source domain images and corresponding labels with a probability of 0.5.

[0057] The training model needs to perform multiple back propagation and parameter updates. Each time the forward model is updated, the same batch of training data is input. Each batch of training data contains n MRI images, n semantic segmentation labels corresponding to the MRI images, and n brain CT images. Among them, n can be determined according to the video memory capacity of the graphics card of the computer when training the model. When n=1, the video memory occupied by training is 4GB. In the present invention, n=1 is selected.

[0058] Before the data is input into the network, it is first matched and aligned (pairing brain tumor MRI images and cerebral hemorrhage CT images with similar semantic content): for each source domain image to be input into the network, the 20 most similar target domain images stored in the similar image selection module in the semantic preservation module in step S12 are searched, one is randomly selected from them, and the two are paired.

[0059] Each time you enter new data, do the following:

[0060] MRI image s is input to the CT image generator G s ,CT image t is input to the MRI generator G t .

[0061] G s (s) and the target domain image t are input to the target domain discriminator D t , G t (t) and the source domain image s are input into the source domain discriminator D s , lock the discriminator update and calculate the generated adversarial loss according to the Wasserstein GAN idea as follows:

[0062]

[0063] in, represents the data extracted from the specified distribution, t is the target domain image, i.e., CT cerebral hemorrhage image, and D t is the target domain image discriminator, G s (s) is the source domain image, i.e., the MRI brain tumor image, which is input into the CT image generator G s The pseudo target domain image synthesized after () is the pseudo CT cerebral hemorrhage image, and p(t) is the distribution of the target domain image; t is extracted from the target domain image. s is the source domain image, i.e., MRI brain tumor image. s is the source domain image discriminator, G t (t) is the target domain image, i.e., the CT cerebral hemorrhage image, which is input into the MRI image generator G t The pseudo source domain image synthesized after () is the pseudo CT image, p(s) is the distribution of the source domain image, It refers to s extracted from the source domain image.

[0064] The cycle consistency loss is calculated as follows:

[0065] s recover =G t (G s (s)) (7)

[0066] t recover =G s (G t (t)) (8)

[0067]

[0068] in, Indicates extracting data from a specified distribution, G s (s), s is consistent with the definition in formula (5), G t (t), t is consistent with the definition in formula (6), s recover The pseudo target domain image, i.e., the pseudo CT image, is input into the CT generator G t () The synthesized cyclic source domain image, i.e., the cyclic synthesized MRI image, t recover The pseudo source domain image, i.e., the pseudo MRI image, is input into the MRI generator G s () is the cyclic target domain image synthesized after synthesis, that is, the cyclic synthesized CT image, p(s) is the distribution of the source domain image, and p(t) is the distribution of the target domain image.

[0069] G s (s) and the label y of the source domain image s s The segmentation subnetwork F is input to the semantics-preserving module t , freeze the update of the segmentation subnetwork and calculate the segmentation loss:

[0070]

[0071] Among them, G s (s) is consistent with the definition in formula (5), y s is the semantic segmentation label of the source domain image, i.e., the MRI image. t The pseudo target domain image segmentation subnetwork in the semantics-preserving module.

[0072] Will s recover and the label y of the source domain image s s The segmentation subnetwork F is input to the semantics-preserving module s ,Freeze the update of the segmentation subnetwork and calculate the segmentation loss:

[0073]

[0074] Among them, s recover Consistent with the definition in formula (7), y s It is consistent with the definition in formula (10). s The recurrent source domain image segmentation subnetwork in the semantics-preserving module.

[0075] The resulting semantic preservation loss is:

[0076]

[0077] in, is the loss calculated by the segmentation subnetwork in (10), is the loss calculated for the segmentation subnetwork in (11).

[0078] The generator loss function is obtained as follows and back-propagated to optimize the generator G s and G t :

[0079]

[0080] in, The generator loss is obtained by integrating the generative adversarial loss in (5) and (6), the cycle consistency loss in (9), and the semantic preservation loss in (12).

[0081] Cut off G s (s), the gradient of S and G s (s) Input semantics-preserving segmentation subnetwork F t , calculate the segmentation loss with the above formula (10), and back propagate the calculated loss to optimize F t

[0082] Cut off S recover The gradient of S and s recover Input semantics-preserving segmentation subnetwork Fs , calculate the segmentation loss with the above formula (11), and back propagate the calculated loss to optimize F s

[0083] Cut off G s The gradient of (s) is calculated as follows and back-propagated to optimize the discriminator D t :

[0084]

[0085] Among them, D t , G s (s) is consistent with the definition in formula (5), y fake is the domain label of the synthetic target domain image, i.e., the CT image, y true is the domain label of the source domain image, i.e., the MRI image.

[0086] The loss is calculated as follows and back-propagated to optimize the discriminator D s :

[0087]

[0088] Among them, D s (s), G t (t) is consistent with the definition in formula (6), y true is the domain label of the source domain image, i.e., the MRI image.

[0089] The above process is repeated until the model meets the preset convergence condition (the preset convergence condition in the invention is that the model runs 100 rounds, and one round refers to a complete training of the model using all the data in the training set).

[0090] S3: Generate training data for training the segmentation network using the data synthesis network trained in step S2 and the labeled source domain image data.

[0091] The source domain image data is preprocessed in the same way as step S2 and input into the generator G in the target domain data synthesis network for prediction. The synthesized target domain data, i.e., the synthesized CT image t synt hetic , and store the resulting images for use as a training set.

[0092] S4: Using the target domain image semantic segmentation network defined in step S1, the training data generated in step S3, and the unlabeled target domain image, train a domain-adaptive image semantic segmentation network.

[0093] First, data preprocessing is performed to convert the data into zero mean and unit variance.

[0094] The source domain image and source domain label are sent to the segmentation network to obtain the segmentation prediction and the intermediate layer upsampling results (sampling to the final predicted segmentation result size, upsampling using bilinear interpolation). The loss is calculated as follows and back-propagated to optimize the segmentation performance of the segmentation network:

[0095]

[0096] Where F is the target domain image semantic segmentation network, and F in (20) h is a subset of F, i.e., the hidden layer h in the semantic segmentation network F (see Figure 3 h1 to h5) and the previous network structure plus an upsampling process, h takes values ​​from h1 to h5. The definition of s is consistent with that in equation (5), y s The definition is consistent with that in formula (10), t synt hetic It is the synthetic target domain image obtained in step S3, that is, the synthetic CT image.

[0097] The target domain image is sent to the segmentation network to obtain segmentation prediction and intermediate layer upsampling results.

[0098] Pause the domain discriminator update, feed the intermediate layer upsampling result into the discriminator, the discriminator outputs the prediction map, calculates the adversarial loss as follows and back-propagates to optimize the generalization performance of the segmentation network:

[0099]

[0100] Among them, F h,tsynt hetic The definition is consistent with that in (20), D is the domain discriminator, and t is defined in the same way as in (6).

[0101] Cut off the gradient update before the source domain and target domain hidden layer representation, feed them into the domain discriminator, calculate the discriminator loss as follows and back propagate to optimize the domain discriminator:

[0102]

[0103] Among them, F h,tsynt hetic The definition is consistent with that in (20), D is the domain discriminator, and t is defined in the same way as in (5).

[0104] The above process is repeated until the model meets the preset convergence condition (the preset convergence condition in the invention is that the model runs 100 rounds, and one round refers to a complete training of the model using all the data in the training set).

[0105] Verification effect:

[0106]

Claims

1. A brain hemorrhage segmentation method based on multi-faceted domain adaptation technology, characterized in that: The following steps are involved: S1: Construct a brain CT image synthesis network based on semantic preservation mechanism and a brain hemorrhage segmentation network based on domain adaptation mechanism; The brain CT image synthesis network based on the semantic preservation mechanism includes two structurally consistent generators, two structurally consistent discriminators, and a semantic preservation module. The semantic preservation module includes two semantic segmentation subnetworks and an image semantically similar image selection module. The generator, It includes feature extraction stage, feature fusion stage and upsampling stage; The feature extraction stage is used to extract and encode features of the preprocessed input image, and the stage includes 4 hidden layers: for the first two hidden layers, each hidden layer includes a 7x7 convolution operation, a layer normalization operation, and a linear rectification function activation operation; for the last two hidden layers, each hidden layer includes a 3x3 convolution operation, a layer normalization operation, and a linear rectification function activation operation; The feature fusion stage re-encodes the feature map generated in the feature extraction stage to integrate the specific semantic information and abstract semantic information under different receptive fields. It includes 6 densely connected modules. The input of the hidden layer in the module adopts dense connection. Each densely connected block includes 5 hidden layers. Each hidden layer contains one step of 3x3 convolution operation, one step of layer normalization operation, and one step of linear rectification function activation operation. Dense connections are defined as: x l+1 =H([x0,…,x l ]) (1) H(x i )=W*σ(B(x i )) (2) d l =d0+g·l (3) Among them, H(x i ) represents the input feature map x i Perform hidden layer operations, [] refers to the cascade operation of feature maps by channel, x i is the feature map of the i-th hidden layer input, l is the ordinal number of the hidden layer, d l represents the number of channels input to the lth hidden layer, g represents the number of channels output by all convolutional layers in the densely connected module, d0 represents the number of channels of the feature map of the first hidden layer of the densely connected module, W represents the convolutional layer weights learned through training, B() represents the layer normalization operation, and σ() represents the linear rectification function activation operation; The upsampling stage completes the conversion from feature encoding to the final output synthetic image, including two hidden layers, each of which contains a 3x3 transposed convolution operation, a layer normalization operation, and a tanh function activation operation; The discriminator comprises a feature extraction module and a prediction module; The feature extraction module includes 5 hidden layers, each of which contains a one-step convolution operation, a one-step layer normalization operation, and a one-step linear rectification function activation operation. The prediction module is a one-step convolution operation, which is used to convert the feature map obtained by the feature extraction module into a prediction map. The segmentation subnetwork of the semantic preservation module includes a pre-trained feature extraction module and a multi-layer upsampling skip-layer connection; the pre-trained feature extraction module is a part of the VGG-16 network pre-trained on the ImageNet dataset, with a total of 5 convolution stages and 5 maximum pooling operations, wherein the first and second convolution stages contain two hidden layers, and the last three convolution stages each contain three hidden layers, each hidden layer contains a 3x3 convolution operation and a linear rectification function activation operation; The two sub-network model architectures are defined in the same way, both of which are pre-trained feature extraction modules combined with multi-layer upsampling skip-layer connection operations; The multi-layer upsampling skip layer connection is defined as follows: The feature maps output by the 5-step max pooling operation of the pre-trained feature extraction module are defined as v_out1, v_out2, v_out3, v_out4, and v_out5 in order; Among them, v_out5 is subjected to deconvolution operation, linear rectification function activation operation, layer normalization operation, and then added to the feature map in v_out4 according to the element value to obtain the feature map temp1; After deconvolution, linear rectification function activation, and layer normalization, temp1 is added to the feature map in v_out3 by element value to obtain feature map temp2. After deconvolution, linear rectification function activation, and layer normalization, temp2 is added to the feature map in v_out2 according to the element value to obtain the feature map temp3; After deconvolution, linear rectification function activation, and layer normalization, temp3 is added to the feature map in v_out1 according to the element value to obtain the feature map temp4; After deconvolution, linear rectification function activation, and layer normalization, temp4 obtains the final feature map. After a 1x1 convolution and a sigmoid activation, the final feature map is thresholded to obtain the predicted segmentation result. The similar image selection module in the semantic retention module is responsible for finding the most similar CT image to each MRI image when it is input. The most similar CT image is randomly selected from the 20 CT images with the highest similarity. The similarity is defined as follows: sim=100d hanming (phash(x),phash(y))+|focus(x)-focus(y)| (4) Where x and y refer to the two images used to calculate the similarity, phash() is an image-aware hashing algorithm that reduces the frequency of the image through discrete cosine transform to obtain the binary fingerprint of the image; d hanming () is the Hamming distance, which is used to obtain the similarity between binary fingerprints of images; focus() calculates the proportion of rough lesion areas in the image. For the source domain image, i.e., MRI image, the proportion of pixels whose semantic segmentation labels are greater than 0 is directly taken as the proportion of all pixels in the image; for the target domain image, i.e., CT image, the image is subjected to threshold binarization operation: for pixel values ​​greater than 120 and less than 180, the value is 1, and the remaining pixel values ​​are binarized as 0; S12: The image semantic segmentation network based on the domain adaptation mechanism includes a segmentation network and a domain discriminator; The segmentation network structure is consistent with the segmentation subnetwork structure in the semantic preservation module of the above-mentioned image synthesis network; The domain discriminator input is the feature map output by the hidden layer and the prediction obtained by the hidden layer after upsampling. It includes 4 hidden layers and one classification layer. Each hidden layer includes a 4x4 convolution operation and a leaky linear rectification function operation. The classification layer is a convolution layer that outputs a prediction map with a dimension of 1. S2: Training the target domain training data synthesis network Furthermore, labeled source domain training data, i.e., MRI images and their semantic segmentation labels, and unlabeled target domain training data, i.e., brain CT images, are required; Preprocess and enhance the training data: Among them, the preprocessing includes sampling the source domain image, source domain label, and target domain image to a consistent size, and normalizing all images, that is, processing the data to zero mean and unit variance; Data augmentation methods: including randomly floating the brightness values ​​of all images by 20%, and horizontally flipping the source domain images and corresponding labels with a probability of 0.5; The training model needs to perform multiple back propagation and parameter updates. Each time the forward model is updated, the same batch of training data is input. Each batch of training data contains n MRI images, n semantic segmentation labels corresponding to the MRI images, and n brain CT images. Select n = 1. Before the data is input into the network, matching alignment is first performed, that is, the brain tumor MRI images and the cerebral hemorrhage CT images with similar semantic contents are paired: for each source domain image to be input into the network, the 20 most similar target domain images stored in the similar image selection module in the semantic preservation module in step S12 are searched, one is randomly selected from them, and the two are paired; Each time you enter new data, do the following: MRI image s is input to the CT image generator G s ,CT image t is input to the MRI generator G t ; G s (s) and the target domain image t are input to the target domain discriminator D t , G t (t) and the source domain image s are input into the source domain discriminator D s , lock the discriminator update and calculate the generated adversarial loss according to the Wasserstein GAN idea as follows: in, represents the data extracted from the specified distribution, t is the target domain image, i.e., CT cerebral hemorrhage image, and D t is the target domain image discriminator, G s (s) is the source domain image, i.e., the MRI brain tumor image, which is input into the CT image generator G s The pseudo target domain image synthesized after () is the pseudo CT cerebral hemorrhage image, and p(t) is the distribution of the target domain image; Let t be extracted from the target domain image; s is the source domain image, i.e., MRI brain tumor image, D s is the source domain image discriminator, G t (t) is the target domain image, i.e., the CT cerebral hemorrhage image, which is input into the MRI image generator G t The pseudo source domain image synthesized after () is the pseudo CT image, p(s) is the distribution of the source domain image, It refers to s extracted from the source domain image; The cycle consistency loss is calculated as follows: s recover =G t (G s (s)) (7) t recover =G s (G t (t)) (8) in, Indicates extracting data from a specified distribution, G s (s), s is consistent with the definition in formula (5), G t (t), t is consistent with the definition in formula (6), s recover The pseudo target domain image, i.e., the pseudo CT image, is input into the CT generator G t () The synthesized cyclic source domain image, i.e., the cyclic synthesized MRI image, t recover The pseudo source domain image, i.e., the pseudo MRI image, is input into the MRI generator G s () is the cyclic target domain image synthesized after synthesis, that is, the cyclic synthetic CT image, p(s) is the distribution of the source domain image, and p(t) is the distribution of the target domain image; G s (s) and the label y of the source domain image s s The segmentation subnetwork F is input to the semantics-preserving module t , freeze the update of the segmentation subnetwork and calculate the segmentation loss: Among them, G s (s) is consistent with the definition in formula (5), y s is the semantic segmentation label of the source domain image, i.e., the MRI image; F t The pseudo target domain image segmentation subnetwork in the semantics preservation module; Will s recover and the label y of the source domain image s s The segmentation subnetwork F is input to the semantics-preserving module s ,Freeze the update of the segmentation subnetwork and calculate the segmentation loss: Among them, s recover Consistent with the definition in formula (7), y s It is consistent with the definition in formula (10); F s The recurrent source domain image segmentation subnetwork in the semantics-preserving module; The resulting semantic preservation loss is: in, is the loss calculated by the segmentation subnetwork in (10), is the loss calculated for the segmentation subnetwork in (11); The generator loss function is obtained as follows and back-propagated to optimize the generator G s and G t : in, The generator loss is obtained by integrating the generative adversarial loss in (5) and (6), the cycle consistency loss in (9), and the semantic preservation loss in (12); Cut off G s (s), the gradient of S and G s (s) Input semantics-preserving segmentation subnetwork F t , calculate the segmentation loss with the above formula (10), and back propagate the calculated loss to optimize F t Cut off recover The gradient of S and s recover Input semantics-preserving segmentation subnetwork F s , calculate the segmentation loss with the above formula (11), and back propagate the calculated loss to optimize F s Cut off G s The gradient of (s) is calculated as follows and back-propagated to optimize the discriminator D t : Among them, D t , G s (s) is consistent with the definition in formula (5), y fake is the domain label of the synthetic target domain image, i.e., the CT image, y true is the domain label of the source domain image, i.e., the MRI image; The loss is calculated as follows and back-propagated to optimize the discriminator D s : Among them, D s (s), G t (t) is consistent with the definition in formula (6), y true is the domain label of the source domain image, i.e., the MRI image; Repeat the above process until the model meets the preset convergence condition, which is running for more than 100 rounds, where one round refers to a complete training of the model using all the data in the training set; S3: Use the data synthesis network trained in step S2 and the labeled source domain image data to generate training data for training the segmentation network; The source domain image data is preprocessed in the same way as step S2 and input into the generator G in the target domain data synthesis network for prediction; the synthesized target domain data, i.e., the synthesized CT image t synthetic , store the obtained images for use as training set; S4: using the target domain image semantic segmentation network defined in step S1, the training data generated in step S3, and the unlabeled target domain image, training a domain-adaptive image semantic segmentation network; First, data preprocessing is performed to process the data into 0 mean and unit variance; The source domain image and source domain label are sent to the segmentation network to obtain the segmentation prediction and the intermediate layer upsampling results, which are sampled to the final predicted segmentation result size. The upsampling adopts bilinear interpolation. The loss is calculated as follows and back-propagated to optimize the segmentation performance of the segmentation network: Where F is the target domain image semantic segmentation network, and F in (20) h is a subset of F, i.e., the hidden layer h in the semantic segmentation network F and the previous network structure plus an upsampling process, where h ranges from h1 to h5; the definition of s is consistent with that in formula (5), and y s The definition is consistent with that in formula (10), t synthetic is the synthetic target domain image obtained in step S3, i.e., the synthetic CT image; Send the target domain image to the segmentation network to obtain segmentation prediction and intermediate layer upsampling results; Pause the domain discriminator update, feed the intermediate layer upsampling result into the discriminator, the discriminator outputs the prediction map, calculates the adversarial loss as follows and back-propagates to optimize the generalization performance of the segmentation network: Among them, F h ,t synthetic The definition is consistent with that in (20), D is the domain discriminator, and t is defined in the same way as in (6); Cut off the gradient update before the source domain and target domain hidden layer representation, feed them into the domain discriminator, calculate the discriminator loss as follows and back propagate to optimize the domain discriminator: Among them, F h ,t synthetic The definition is consistent with that in (20), D is the domain discriminator, and t is defined in the same way as in (5); Repeat the above process until the model meets the preset convergence condition; the preset convergence condition is to run more than 100 rounds, and one round means that the model is fully trained once using all the data in the training set.

Images