Medical image segmentation method based on multi-modal self-supervision
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEBEI UNIV OF TECH
- Filing Date
- 2022-11-04
- Publication Date
- 2026-07-10
AI Technical Summary
Existing self-supervised learning methods struggle to effectively utilize the modal characteristics and common knowledge of multimodal images in medical image segmentation tasks, resulting in poor segmentation performance, especially when labeled data is scarce.
A multimodal self-supervised approach is adopted. By constructing a cyclic consistency modality contrastive translation network, cross-domain translation and adversarial training are performed using a generator and a discriminator to learn the semantic consistency and modal characteristics of multimodal images. The weight transfer of the generator is combined to provide initial weights for the segmentation network.
It improves the segmentation performance of medical image segmentation networks with limited labeled data, enabling them to adapt more flexibly to clinical environments, provide more comprehensive segmentation results, reduce labeling costs and time, and improve segmentation accuracy.
Smart Images
Figure CN115601352B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image segmentation technology, specifically a medical image segmentation method based on multimodal self-supervised methods. Background Technology
[0002] The tremendous success of fully supervised deep learning methods in segmentation and classification tasks relies heavily on a large amount of labeled data. However, due to the scarcity of labeled medical images, fully supervised deep learning methods struggle to achieve ideal results in medical image segmentation and classification. Self-supervised learning, on the other hand, first learns from a large amount of unlabeled data and then fine-tunes the model parameters using a small amount of labeled data. Therefore, self-supervised learning has lower requirements for the amount of labeled data and can address the problem of scarce labeled medical images.
[0003] Current self-supervised learning methods applied in the field of medical imaging can be broadly categorized into two types. One type is based on restoration paradigms, such as rotation restoration, scrambled image restoration, and image completion. These methods learn general structural knowledge of images. The other type is based on contrastive learning. This type of method requires training an encoder to generate paired image representations for different augmented views of the input image and defining them as positive pairs. Augmented views from different images are defined as negative pairs. The goal is to maximize the similarity of positive pairs in the input image while minimizing the similarity of negative pairs, thereby learning general image semantic information. However, most of these methods utilize unimodal images and rarely consider the important multimodal images in medical imaging. Existing methods that consider multimodal images mainly learn general multimodal structural information through multimodal mosaic restoration, ignoring useful modal characteristic information, or they use contrastive learning frameworks to learn semantic relationships through semantic alignment of multimodal information. However, simply performing semantic alignment inevitably leads to the loss of a large amount of useful semantic information, which is detrimental to downstream image segmentation tasks with dense prediction.
[0004] For the reasons mentioned above, this invention proposes a medical image segmentation method based on multimodal self-supervision. By using self-supervised pre-training, the upstream network can fully learn modal characteristics and common modal knowledge, which is beneficial to improving the segmentation performance of the downstream segmentation network. Summary of the Invention
[0005] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a medical image segmentation method based on multimodal self-supervision.
[0006] The technical solution adopted by this invention to solve the aforementioned technical problem is: to provide a medical image segmentation method based on multimodal self-supervision, specifically including the following steps:
[0007] Step 1: Acquire multimodal medical images of a specific lesion tissue, including modality A and modality B images; preprocess all images before saving;
[0008] Step 2: Construct a cycle-consistent modal contrastive translation network, including the generator G. A→B Generator G B→A Discriminator A and discriminator Dis B Generator G A→B Including encoder E A Intermediate shared module and decoder D A Generator G B→A Including encoder E B Intermediate shared module and decoder D B Two encoders are connected to the same intermediate shared module to achieve weight sharing;
[0009] Step 3: Pre-train the Cyclic Consistency Modal Contrast Domain Translation Network, design a loss function to calculate the training loss, and obtain the network weights after pre-training; the loss function includes multimodal semantic consistency loss, adversarial loss, cross-domain translation loss, and cycle consistency loss;
[0010] First, pairs of A-modal images and B-modal images are input into their respective generators for cross-domain generation, yielding feature F extracted from the A-modal image. A and features F extracted from the B modality image B The image includes a cross-modal image B' derived from modal image A and a cross-modal image A' derived from modal image B; similarly, the corresponding features F are extracted from cross-modal images A' and B'. A' and F B' According to feature F A and F B And calculate the multimodal semantic consistency loss for cross-modal images A' and B'. The calculation formula is:
[0011]
[0012]
[0013] In the formula, A i and B i Let l represent a pair of two modal images, and l(.) represent the multimodal semantic consistency loss between the pair of images. These represent the images from image A. i and B i The features extracted are t, which represents the temperature hyperparameter, sim(.) which represents the cosine similarity between two features, and Γ. +Γ represents the set of all paired modal images. - Represents the relationship with image A i A set of dissimilar images, F z Represents set Г - The image in, A' i and B' i Γ' represents a pair of cross-modal images. + Let N represent the set of all paired cross-modal images, where i∈N represents the image number and N represents the total number of paired images in the dataset;
[0014] Simultaneously, adversarial loss and cross-domain translation loss are calculated using cross-modal images A' and B', respectively. The adversarial loss helps the generator learn the domain distribution from one modality to another, while simultaneously training the discriminator to correctly identify the source of its own input. The formula for calculating the adversarial loss is:
[0015]
[0016] In the formula, Let A and B represent the expected values of given distributions for modality A and modality B, respectively.
[0017] Cross-domain translation loss is used to help the generator better learn the detailed features of cross-modal images. The formula for calculating cross-domain translation loss is:
[0018]
[0019] In the formula, ||.||1 represents the 1-norm;
[0020] Then, the cross-modal images A' and B' are input into the generator G respectively. A→B and G B→A Cross-domain reconstruction is performed to obtain reconstructed modeling images A'' and B'', and the cycle consistency loss is calculated using the reconstructed modeling images A'' and B''. The expression is:
[0021]
[0022] In summary, the loss function for network training is expressed as:
[0023]
[0024] Where γ, β, and λ all represent coefficient hyperparameters;
[0025] Step 4: Construct a medical image segmentation network, including an A-modal segmentation network and a B-modal segmentation network. Both the A-modal and B-modal segmentation networks are isomorphic to the generator of the cycle-consistent modality contrast domain translation network; the generator G... A→B encoder EA The weights of the intermediate shared modules are transferred to the A-mode segmentation network as initial weights for the A-mode segmentation network; the generator G is then transferred to the A-mode segmentation network. B→A encoder E B The weights of the intermediate shared module are transferred to the B-modal segmentation network as the initial weights of the B-modal segmentation network; the A-modal segmentation network and the B-modal segmentation network are trained respectively, and the trained A-modal segmentation network and B-modal segmentation network are used for medical image segmentation of the corresponding modalities respectively.
[0026] Furthermore, the two generators have the same network structure. The encoder includes three convolutional layers. The output of the first convolutional layer is batch normalized and activated before being input into the second convolutional layer. The first convolutional layer has a kernel size of 3*3, a stride of 1, and 32 kernels; the second convolutional layer has a kernel size of 3*3, a stride of 2, and 64 kernels; and the third convolutional layer has a kernel size of 3*3, a stride of 2, and 128 kernels. The intermediate shared module includes two convolutional layers and one upsampling layer. The first convolutional layer has a kernel size of 3*3, a stride of 2, and 256 kernels; the second convolutional layer has a kernel size of 3*3, a stride of 2, and 256 kernels; and the second convolutional layer has a kernel size of 3*3, a stride of 2, and 128 kernels. The convolutional kernel size is 3*3, stride is 2, and number of kernels is 512. The convolutional kernel size of the upsampling layer is 2*2, stride is 2, and number of kernels is 256. The decoder consists of three upsampling layers and one convolutional layer. The output of the convolutional layer is batch normalized and activated to obtain the output of the generator. The first upsampling layer has a convolutional kernel size of 2*2, stride is 2, and number of kernels is 128. The second upsampling layer has a convolutional kernel size of 2*2, stride is 2, and number of kernels is 64. The third upsampling layer has a convolutional kernel size of 2*2, stride is 2, and number of kernels is 32. The convolutional sum size of the convolutional layer is 3*3, stride is 1, and number of kernels is 1.
[0027] The two discriminators have the same network structure, each consisting of five convolutional layers and one global pooling layer. The output of each convolutional layer is batch normalized and activated before being input into the next convolutional layer. The first convolutional layer has a kernel size of 3*3, a stride of 2, and 64 kernels. The second convolutional layer has a kernel size of 3*3, a stride of 2, and 128 kernels. The third convolutional layer has a kernel size of 3*3, a stride of 2, and 256 kernels. The fourth and fifth convolutional layers both have a kernel size of 3*3, a stride of 2, and 512 kernels. The global pooling layer has 1 kernel.
[0028] Compared with the prior art, the beneficial effects of the present invention are:
[0029] 1. Using comparative cross-domain translation as a multimodal self-supervised pre-training task to learn more comprehensive modal features can improve the segmentation results of each modality when only a small amount of labeled data is available, allowing for more flexible adaptation to clinical environments. While theoretically the segmentation results for each modality should be the same, in practice, the segmentation prediction results for different modalities often differ. This is because different modalities have their own advantages in reflecting the physical characteristics of organs / tissues / lesions. For example, certain tissues or tumors are better visualized in PET modality, while CT modality better reflects bones, providing more accurate location information. Therefore, the segmentation prediction results of multiple modalities for a patient can complement each other, providing doctors with a more comprehensive reference. On the other hand, due to the scarcity of medical equipment, if patients need to wait to obtain medical images of all modalities or require specific modal images for segmentation prediction, the wait may be very long, delaying treatment. When multimodal images cannot be obtained simultaneously, patients can obtain images of any modality for segmentation prediction, allowing doctors to promptly formulate subsequent treatment plans based on the prediction results.
[0030] 2. This invention utilizes a Cyclic Consistent Modal Contrast Domain Translation Network for multimodal self-supervised pre-training, fully learning modal characteristics and common modal knowledge. The Cyclic Consistent Modal Contrast Domain Translation Network incorporates an intermediate sharing module and multimodal semantic consistency loss into the traditional Cyclic Consistent Cross-Domain Translation Network, promoting better learning of modal characteristics and common modal knowledge to generate higher-quality pre-trained network weights. This provides better initial weights for the segmentation network, improving its segmentation ability when only a limited amount of labeled data is available. Compared to other self-supervised methods for medical image segmentation, this invention more comprehensively and fully mines the rich multimodal information of medical images, achieving mutual complementarity of multimodal information and promoting more accurate segmentation results. Using the initial network weights obtained through the multimodal self-supervised method of this invention in a fully supervised segmentation model, followed by fine-tuning with a small amount of labeled data, significantly improves segmentation accuracy. This demonstrates that the method proposed in this application can effectively learn and mine beneficial information from unlabeled multimodal medical image data, reduce model label dependence, and improve the segmentation ability of the segmentation network when only a limited amount of labeled data is available.
[0031] 3. By fully mining the beneficial information of a large amount of unlabeled multimodal image data, this invention greatly reduces the number of labels required for training the network and the labeling cost while ensuring the network segmentation performance. Attached Figure Description
[0032] Figure 1 This is the overall flowchart of the present invention. Detailed Implementation
[0033] The technical solution of the present invention will be described in detail below with reference to the embodiments and accompanying drawings, but this is not intended to limit the scope of protection of this application.
[0034] This invention provides a multimodal self-supervised medical image segmentation method (hereinafter referred to as the method), comprising the following steps:
[0035] Step 1: Acquire multimodal medical images of a certain lesion tissue, including A-modal images and B-modal images; preprocess all images: first, resample all images to a uniform pixel density, then perform grayscale value normalization, and finally process all images to a uniform size and cut them into 2D slices according to the interlayer position for saving.
[0036] Step 2: Construct a cyclically consistent modal contrastive translation network, which includes a generator G. A→B Generator G B→A Discriminator A and discriminator Dis B Generator G A→B An encoder E is used to convert an A-modal image into a B-modal image, including encoding the A-modal image. A The intermediate shared module SL and the decoder D, which can decode features into B-modal images. A Generator G B→A An encoder E is used to convert a B-modal image into an A-modal image, including encoding the B-modal image. B The intermediate shared module SL and the decoder D that decodes features into an A-modal image. B The generator's role is to perform cross-domain translation on the input image and generate a cross-modal image, fully learning modal features through cross-domain translation. Two encoders are connected to the same intermediate shared module to achieve weight sharing. The discriminator is used to determine the source of its own input and optimize the generator's cross-domain translation performance. A Used to determine whether its input is a real A-modal image or a generator G. A→B Cross-domain generated A-modal image, discriminator Dis B Used to determine whether its input is a real B-modal image or a generator G. B→A Cross-domain generated B-modal images;
[0037] The two generators have the same network structure. The encoder includes three convolutional layers. The output of the first convolutional layer is batch normalized and activated before being input into the second convolutional layer. The first convolutional layer has a kernel size of 3*3, a stride of 1, and 32 kernels; the second convolutional layer has a kernel size of 3*3, a stride of 2, and 64 kernels; and the third convolutional layer has a kernel size of 3*3, a stride of 2, and 128 kernels. The intermediate shared module includes two convolutional layers and one upsampling layer. The first convolutional layer has a kernel size of 3*3, a stride of 2, and 256 kernels; the second convolutional layer has a kernel size of 3*3, a stride of 2, and 256 kernels. The size of the convolutional kernels in the upsampling layer is 3*3, the stride is 2, and the number of kernels is 512. The kernel size of the upsampling layer is 2*2, the stride is 2, and the number of kernels is 256. The decoder consists of three upsampling layers and one convolutional layer. The output of the convolutional layer is batch normalized and activated to obtain the output of the generator. The kernel size of the first upsampling layer is 2*2, the stride is 2, and the number of kernels is 128. The kernel size of the second upsampling layer is 2*2, the stride is 2, and the number of kernels is 64. The kernel size of the third upsampling layer is 2*2, the stride is 2, and the number of kernels is 32. The convolutional sum of the convolutional layer is 3*3, the stride is 1, and the number of kernels is 1.
[0038] The two discriminators have the same network structure, each consisting of five convolutional layers and one global pooling layer. The output of each convolutional layer is batch normalized and activated before being input into the next convolutional layer. The first convolutional layer has a kernel size of 3*3, a stride of 2, and 64 kernels. The second convolutional layer has a kernel size of 3*3, a stride of 2, and 128 kernels. The third convolutional layer has a kernel size of 3*3, a stride of 2, and 256 kernels. The fourth and fifth convolutional layers both have a kernel size of 3*3, a stride of 2, and 512 kernels. The global pooling layer has 1 kernel.
[0039] Step 3: Pre-train the Cyclic Consistency Modal Contrastive Domain Translation Network, calculate the network training loss using the loss function, and obtain the network weights after pre-training; the loss function includes multimodal semantic consistency loss, adversarial loss, cross-domain translation loss, and cycle consistency loss;
[0040] During network forward training, N pairs of images from two modalities are randomly sampled to form the dataset. Image A i With B i For alignment, image A i With B j For negative pairs (i≠j); first, the paired A-mode image and B-mode image (the images of the two modes are taken from the same lesion tissue) are input into their respective generators for cross-domain generation, that is, the A-mode image is input into generator G. A→B In the process, the B-modal image is input into the generator G. B→AIn this process, the feature F extracted from the A modality image is obtained. A and features F extracted from the B modality image B The image includes a cross-modal image B' derived from modal image A and a cross-modal image A' derived from modal image B; similarly, the corresponding features F are extracted from cross-modal images A' and B'. A' and F B' ;
[0041] Multimodal semantic consistency loss helps to narrow the distance between paired modal semantics while widening the distance between unpaired modal semantics. This promotes the network's learning of multimodal semantic consistency information, enabling the recurrent consistent modal contrastive domain translation network to learn both modal characteristic knowledge and shared modal knowledge. This provides high-quality pre-trained network weights for the segmentation network, thereby improving its segmentation performance when only a small amount of labeled data is available. The calculation formula is:
[0042]
[0043]
[0044] In the formula, A i and B i Let l represent a pair of two modal images, and l(.) represent the multimodal semantic consistency loss between the pair of images. These represent the images from image A. i and B i The features extracted are t, which represents the temperature hyperparameter, sim(.) which represents the cosine similarity between two features, and Γ. + Γ represents the set of all paired modal images. - Represents the relationship with image A i A set of dissimilar images, F z Represents set Г - The image in, A' i and B' i Γ' represents a pair of cross-modal images. + This represents the set of all paired cross-modal images;
[0045] Simultaneously, adversarial losses are calculated using cross-modal images B' and A'. and cross-domain translation loss Adversarial loss is used to help the generator G A→B Learn the domain distribution from mode A to mode B, and help the generator G. B→AThe generator learns the domain distribution from mode B to mode A, enabling it to generate realistic cross-domain images, while simultaneously training the discriminator to correctly identify the source of its own input; the adversarial loss is calculated as follows:
[0046]
[0047] in, Let A and B represent the expected values of given distributions for modality A and modality B, respectively.
[0048] Cross-domain translation loss help generator G A→B and G B→A To better learn the detailed features of cross-modal images, for modal image A, the input to generator G A→B Cross-domain generation is performed to obtain a cross-modal image B'. The cross-domain translation loss constrains the cross-modal image B' generated after cross-domain generation to be as similar as possible to the original input B modal image at the pixel level, which helps the generator learn more details at the pixel level; the same applies to the B modal image; the cross-domain translation loss is calculated as follows:
[0049]
[0050] Where ||.||1 represents the 1-norm;
[0051] Then, the cross-modal images B' and A' are input into the generator G, respectively. B→A and G A→B Cross-domain reconstruction is performed to obtain not only the features F extracted from the cross-modal image B'. B‘ and F extracted from cross-modal image A' A‘ Furthermore, reconstructed modeling images A'' and B'' are obtained; using the reconstructed modeling images A'' and B'', the cycle consistency loss is calculated according to equation (5). Cycle consistency loss helps the generator learn more accurate one-to-one bidirectional cross-domain mappings, thereby promoting the generator to learn better image features; for example, an image of modality A is processed by generator G. A→B The cross-modal image B' generated across domains is fed into the generator G. B→A Cross-domain reconstruction is performed to obtain the reconstructed modal image A”. Cycle consistency loss makes the reconstructed modal image A” obtained after two cross-domain generations as similar as possible to the original input modal image A.
[0052]
[0053] In summary, the loss function for network training is expressed as:
[0054]
[0055] Where γ, β, and λ all represent coefficient hyperparameters;
[0056] Step 4: Construct a medical image segmentation network, including an A-modal segmentation network and a B-modal segmentation network. Both the A-modal and B-modal segmentation networks are isomorphic to the generator of the Cyclic Consistency Modality Contrast Domain Translation Network. The weights of the pre-trained Cyclic Consistency Modality Contrast Domain Translation Network are then transferred to the segmentation network, i.e., the generator G... A→B encoder E A The weights of the intermediate shared module SL are transferred to the A-mode segmentation network as the initial weights of the A-mode segmentation network; the generator G is... B→A encoder E B The weights of the intermediate shared module SL are transferred to the B-modal segmentation network as the initial weights of the B-modal segmentation network. Then, the A-modal segmentation network and the B-modal segmentation network are trained with a small amount of labeled data respectively. The trained A-modal segmentation network and B-modal segmentation network are then used for medical image segmentation of the corresponding modalities.
[0057] Example 1
[0058] This example utilizes the method described above to perform tumor segmentation with minimal annotation on CT-PET multimodal medical images from a head and neck cancer tumor database. The head and neck cancer tumor database includes 201 cases, each with 3D medical images in both CT and PET modalities and corresponding pixel-level annotations; it includes the following:
[0059] First, data preprocessing is performed: CT-PET multimodal medical images are acquired from the head and neck cancer tumor database. Each case is placed in a folder named after its ID, and each folder stores the CT and PET modal images of that case, along with their corresponding pixel-level segmentation and diagnostic results. All images are resampled to a uniform pixel density, and then grayscale normalization is performed. Specifically, based on the image's grayscale value range [a, b], the grayscale value range of the image is calculated as ba. If the grayscale value of a pixel in the image is x, then the normalized grayscale value of that pixel is... Finally, all images are processed to a size of 144*144*144 pixels and then cut into 2D slices along the Z-axis according to the interlayer position for storage.
[0060] Dataset partitioning: Each dataset is randomly divided into a training set and a test set. The test set is not used for pre-training of the Cyclic Consistency Modality Contrast Domain Translation Network or training of the segmentation task; it is only used for evaluation. For the head and neck cancer tumor dataset, the training set contains images of 180 patients, and the test set contains images of 21 patients. During the pre-training phase of the Cyclic Consistency Modality Contrast Domain Translation Network, the labeled data in the training set is not used. During the segmentation task phase of the segmentation network, 10% of the labeled data in the training set is randomly selected to fine-tune the segmentation network for each modality.
[0061] Then, a cyclic consistency mode contrast domain translation network is constructed according to step two, and the network is trained according to step three. The training process is constrained by the total loss of the network training to obtain the pre-trained cyclic consistency mode contrast domain translation network.
[0062] Finally, a segmentation network is constructed, including an A-mode segmentation network and a B-mode segmentation network. Both the A-mode and B-mode segmentation networks are isomorphic to the generator of the Cyclic Consistency Mode Contrast Translation Network. The weights of the pre-trained Cyclic Consistency Mode Contrast Translation Network are then transferred to the segmentation network, i.e., the generator G... A→B encoder E A The weights of the intermediate shared module SL are transferred to the A-mode segmentation network as the initial weights of the A-mode segmentation network; the generator G is... B→A encoder E B The weights of the intermediate shared module SL are transferred to the B-modal segmentation network as the initial weights of the B-modal segmentation network. 10% of the labeled data in the training set are randomly selected to train the A-modal segmentation network and the B-modal segmentation network. The trained A-modal segmentation network and B-modal segmentation network are then used for medical image segmentation of the corresponding modalities to produce high-precision segmentation results.
[0063] The segmentation results were evaluated using DICE as the evaluation metric. DICE represents the proportion of the segmented region that intersects with the ground truth region, with a value ranging from 0 to 1. A higher value indicates a more accurate segmentation result, and a DICE value of 1 indicates that the segmented region completely overlaps with the ground truth region. When only 10% of the labeled data was used, the DICE value for CT modality segmentation was 0.3442, which is 9.01 percentage points higher than the DICE value of 0.2541 obtained by the same segmentation network without pre-training for CT modality image segmentation; the DICE value for PET modality segmentation was 0.6488, which is 7.19 percentage points higher than the DICE value of 0.5769 obtained by the same segmentation network without pre-training for PET modality image segmentation. Therefore, the multimodal self-supervised pre-training effect of this invention is significant.
[0064] Any aspects not covered in this invention are applicable to existing technologies.
Claims
1. A medical image segmentation method based on multimodal self-supervised analysis, characterized in that, The method includes the following steps: Step 1: Acquire multimodal medical images of the lesion tissue, including Modal images and Modal images; all images are preprocessed before being saved; Step 2: Construct a cycle-consistent modal contrastive translation network, including a generator. Generator Discriminator and discriminator generator Including encoders Intermediate shared module and decoder generator Including encoders Intermediate shared module and decoder The two encoders are connected to the same intermediate shared module to achieve weight sharing; Step 3: Pre-train the Cyclic Consistency Modal Contrast Domain Translation Network, design a loss function to calculate the training loss, and obtain the network weights after pre-training; the loss function includes multimodal semantic consistency loss, adversarial loss, cross-domain translation loss, and cycle consistency loss; First, pair up Modal images and Modal images are input into their respective generators for cross-domain generation, resulting in images derived from... Features extracted from modal images and from Features extracted from modal images and by Cross-modal images obtained from modal image conversion and by Cross-modal images obtained from modal image conversion Similarly, from cross-modal images and Extract the corresponding features and Based on characteristics and and features and Calculate the multimodal semantic consistency loss The calculation formula is: （1） （2） In the formula, and This represents a pair of modal images. This represents the multimodal semantic consistency loss between pairs of images. , Representing from the image and Features extracted from This indicates temperature hyperparameters. This represents the cosine similarity between two features. This represents the set of all paired modal images. Representation and Image Dissimilar image sets Represents a set The image in and This represents a pair of cross-modal images. This represents the set of all paired cross-modal images. Indicates the image number. This represents the total number of paired images in the dataset; Simultaneously, utilizing cross-modal images and The adversarial loss and cross-domain translation loss are calculated separately. The adversarial loss helps the generator learn the domain distribution from one modality to another, while training the discriminator to correctly identify the source of its own input. The formula for calculating the adversarial loss is: （3） In the formula, , They represent respectively to Modal images and Calculate the expectation of a given distribution of a modal image; Cross-domain translation loss is used to help the generator better learn the detailed features of cross-modal images. The formula for calculating cross-domain translation loss is: （4） In the formula, Represents the 1-norm; Then, the cross-modal image and Input into the generator respectively and Cross-domain reconstruction is performed to obtain the reconstructed model morphological image. and and utilize remodeled state images and Calculate cycle consistency loss The expression is: （5） In summary, the loss function for network training is expressed as: （6） in, , , All represent coefficient hyperparameters; Step 4: Construct a medical image segmentation network, including an A-modal segmentation network and a B-modal segmentation network. Both the A-modal and B-modal segmentation networks are isomorphic to the generator of the cycle-consistent modality contrast domain translation network; the generator... encoder The weights of the intermediate shared modules are transferred to the A-mode segmentation network as initial weights for the A-mode segmentation network; the generator... encoder The weights of the intermediate shared module are transferred to the B-modal segmentation network as the initial weights of the B-modal segmentation network; the A-modal segmentation network and the B-modal segmentation network are trained respectively, and the trained A-modal segmentation network and B-modal segmentation network are used for medical image segmentation of the corresponding modalities respectively.
2. The medical image segmentation method based on multimodal self-supervised analysis according to claim 1, characterized in that, The two generators have the same network structure. The encoder consists of three convolutional layers. The output of the first convolutional layer is batch normalized and activated before being fed into the second convolutional layer. The kernel size of the first convolutional layer is [missing information]. The stride is 1, the number of convolutions is 32, and the kernel size of the second convolutional layer is [missing value]. The stride is 2, the number of convolutions is 64, and the kernel size of the third convolutional layer is [missing value]. The stride is 2, and the number of convolutional layers is 128. The intermediate shared module consists of two convolutional layers and one upsampling layer. The kernel size of the first convolutional layer is... The stride is 2, the number of convolutions is 256, and the kernel size of the second convolutional layer is [missing value]. The stride is 2, the number of convolutions is 512, and the kernel size of the upsampling layer is [missing value]. The stride is 2, and the number of steps is 256. The decoder consists of three upsampling layers and one convolutional layer. The output of the convolutional layer is batch normalized and activated to obtain the generator's output. The kernel size of the first upsampling layer is [missing value]. The stride is 2, the number of convolutions is 128, and the kernel size of the second upsampling layer is [missing value]. The stride is 2, the number of convolutions is 64, and the kernel size of the third upsampling layer is [missing value]. With a stride of 2 and a number of 32, the convolutional sum of the convolutional layers is [value missing]. The step size is 1, and the number of steps is 1. The two discriminators have the same network structure, each consisting of five convolutional layers and one global pooling layer. The output of each convolutional layer is batch normalized and activated before being fed into the next convolutional layer. The kernel size of the first convolutional layer is [size missing]. The stride is 2, the number of convolutions is 64, and the kernel size of the second convolutional layer is [missing value]. The stride is 2, the number of convolutions is 128, and the kernel size of the third convolutional layer is [missing value]. The stride is 2, the number of convolutions is 256, and the kernel size of the fourth and fifth convolutional layers is 1. The stride is 2, the number of convolution kernels is 512, and the number of convolution kernels in the global pooling layer is 1.