A multimodal medical image segmentation system

By performing cross-modal image generation and comparison learning in a multimodal medical image segmentation system, combining dynamic weight control and region-aware constraints, the spatial offset and modal difference problems of unpaired images are solved, and the segmentation performance of the model is improved. Remarkable results were achieved especially in the cardiac substructure segmentation task.

CN118864864BActive Publication Date: 2024-11-26JILIN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411337970.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-25
Publication Date
2024-11-26
Estimated Expiration
2044-09-25

AI Technical Summary

Technical Problem

In clinical environments, unpaired and unlabeled multimodal medical images are difficult to be used as simultaneous input of models due to spatial dysregulation and modal differences, limiting model performance, and existing unsupervised domain adaptive methods have failed to effectively solve this question.

Method used

A multimodal medical image segmentation system was designed, and cross-modal image generation and comparison learning were performed through the image generation module to obtain registered images, and the segmentation operation was performed using the Unet network in the segmentation module, combining dynamic weight control mechanism and region Perceptual constraints enable seamless integration of image translation and segmentation stages.

Benefits of technology

The spatial dysfunction and modal differences of unpaired data are effectively solved, the segmentation performance of the model is improved, especially in the cardiac substructure segmentation task, and the dynamic weight control mechanism improves the flexibility of the training process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118864864B_ABST
    Figure CN118864864B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimodal medical image segmentation system, which belongs to the technical field of medical image segmentation. The system includes: an image generation module and a segmentation module; the image generation module is used to perform cross-modal image generation on non-paired images to obtain a generated image; the generated image is subjected to comparative learning to obtain a registered image; the segmentation module is used to perform a segmentation operation on the input registered image and the source image using a segmentation model to obtain a segmented image. In order to solve the problem that the traditional static weight allocation method lacks flexibility in a dynamic environment and cannot be adaptively adjusted throughout the training process to reflect changes in task importance, the present invention proposes a dynamic weight control mechanism.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image segmentation, and in particular relates to a multimodal medical image segmentation system. Background Art

[0002] Multimodal medical image segmentation is crucial to improve the accuracy of diagnosis in various clinical settings. However, it is difficult to obtain complete data in actual clinical settings, which severely limits the use of unpaired multimodal data. The reason is that multimodal unpaired images have spatial misalignment and modality differences, which not only increase the richness of image information but also increase the interference to the target task. The feature extractor is therefore unable to integrate information from different modalities, which affects the model performance. In response to the problems of spatial misalignment and modality differences, some studies have attempted to use unsupervised domain adaptation methods to align the distribution of the source modality and the target modality to alleviate the domain shift problem. At present, there are two main ideas: one is image alignment and the other is feature alignment. These methods mainly focus on solving the difficulties of modality information misalignment caused by spatial misalignment and morphological differences. The evaluation of segmentation tasks focuses more on the accurate identification and positioning of specific structures or regions in the image. From this perspective, existing methods based on unsupervised domain adaptation alleviate the problems of misalignment and mispairing between different modalities, but these methods do not bring significant improvements to the segmentation effect of source domain images. In addition, the visual evaluation of the modality-converted image is not necessarily positively correlated with the evaluation of the segmentation task. Even if the image is not visually optimal, it is valuable to the segmentation algorithm as long as it can accurately highlight the boundaries and features of the target structure. Summary of the invention

[0003] The present invention aims to solve the problem in the prior art that it is difficult to obtain complete data in actual clinical environments, the use of unpaired and unlabeled multimodal data is severely limited, and the unpaired data cannot be used as simultaneous input of the model due to spatial imbalance and morphological differences. A multimodal medical image segmentation system is proposed to seamlessly integrate the image translation and segmentation stages.

[0004] To achieve the above object, the present invention provides the following solution: a multimodal medical image segmentation system, comprising: an image generation module and a segmentation module;

[0005] The image generation module is used to perform cross-modal image generation on unpaired images to obtain generated images; the generated images are subjected to comparative learning to obtain registered images;

[0006] The segmentation module is used to perform a segmentation operation on the input registration image and the source image using a segmentation model to obtain a segmented image.

[0007] Further preferably, the image generation module uses a translation network to perform a cross-modal transformation operation on the unpaired image to obtain the generated image;

[0008] The translation network includes two adversarial learning networks with the same structure; each adversarial learning network includes a shared encoder, a translation decoder, a discriminator and an auxiliary segmentation decoder;

[0009] The source image passes through the shared encoder and the translation decoder to obtain the generated image of a different modality from the source image; the generated image and a real image of the same modality as the generated image are input into the discriminator to make the generated image approximate the real image of the same modality as the generated image; the source image passes through the shared encoder and the auxiliary segmentation decoder to output the predicted label of the source image, and the predicted label and the real label of the source image are used to calculate the mixed segmentation loss to complete the segmentation constraint.

[0010] Further preferably, the total loss of the translation network includes:

[0011] ,

[0012] Where, L ADV Indicates resistance loss; L SC represents the region-aware segmentation loss; L CL represents the contrast consistency loss; λ1, λ2 and λ3 are weight coefficients.

[0013] Further preferably, a dynamic weight control mechanism is used to adjust the weight coefficient;

[0014] For λ1, the adjustment mechanism is as follows:

[0015] ,

[0016] Where f(e) is the nonlinear attenuation function; σ is a hyperparameter used to adjust the attenuation rate; N ep Indicates the number of training rounds epoch;

[0017] For λ2 and λ3, the adjustment mechanism is as follows:

[0018] ,

[0019] ,

[0020] In the formula, Represents the loss function The gradient of the parameter vector θ; θ i represents the i-th parameter in the parameter vector θ.

[0021] Further preferably, the countermeasure loss includes:

[0022] ,

[0023] In the formula, and Denote the real image and the generated image respectively; D syn represents the discriminator.

[0024] Further preferably, the region-aware segmentation loss includes:

[0025] ,

[0026] Where, L Dice represents Dice loss; L CE represents the cross entropy loss; and denote the source image label and the predicted label respectively.

[0027] Further preferably, the comparison consistency loss includes:

[0028] ,

[0029] In the formula, and represents contrast loss;

[0030] in,

[0031] ,

[0032] ,

[0033] In the formula, Represents all possible sample sets { , ,…… }Calculate the expected value in [ ]; represents a fixed ma modal sample image; It represents samples with the same semantics obtained by converting a fixed ma modality image through an adversarial learning network to the mb modality; Yes Apply three different random data augmentation transformations to obtain multiple samples with the same modality but different semantics; ( )and ( ) represent two encoders respectively; τ represents the temperature coefficient.

[0034] Further preferably, the segmentation model adopts a Unet network; the loss of the Unet network includes:

[0035] ,

[0036] Where, L Dice represents Dice loss; L CE represents the cross entropy loss; p, y represent the predicted labels of the segmentation network and the true labels of the source images, respectively.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) In order to solve the problem that the traditional static weight allocation method lacks flexibility in a dynamic environment and cannot be adaptively adjusted to reflect the changes in task importance during the entire training process, a dynamic weight control mechanism is proposed in this paper.

[0039] (2) The present invention designs a region-aware constraint L SC to align visual quality with segmentation utility.

[0040] (3) The present invention focuses on modality-invariant structural features and aligns source domain features with target domain features from an independent perspective of modality-specific information. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A schematic diagram of a process of translating a network to obtain a registered image according to an embodiment of the present invention;

[0043] Figure 2 It is a schematic diagram of a process of obtaining the registered image through comparative learning in an embodiment of the present invention;

[0044] Figure 3 Schematic diagram of the structure of the image segmentation stage in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Embodiment 1:

[0048] The present embodiment provides a multimodal medical image segmentation system, including: an image generation module and a segmentation module; the image generation module is used to perform cross-modal image generation on an unpaired image to obtain a generated image; the generated image is subjected to comparative learning to obtain a registered image; the segmentation module is used to operate the input registered image and the source image using a segmentation model to obtain a segmented image.

[0049] In this embodiment, the image generation module uses a translation network to obtain a generated image. Specifically, two translation networks are used to convert an image in the ma mode into an image in the corresponding mb mode, and to convert an image in the mb mode into an image in the corresponding ma mode, where ma and mb represent two different modes, respectively, to minimize pixel-level structural deviation. The process is as follows: Figure 1 As shown. The translation network includes two adversarial learning networks with the same structure; each adversarial learning network includes a shared encoder E syn , a corresponding translation decoder U syn , a discriminator D for controlling the quality of generated images syn and an auxiliary segmentation decoder U for segmentation constraints seg ; During the training process, each adversarial learning network corresponds to a modality, and the standard adversarial loss L ADV To coordinate the shared encoder E syn and the translation decoder U syn Training of the discriminator D syn It is used to distinguish between real images and generated images. In the decoding process of image translation, an auxiliary segmentation decoder U is deployed seg To explain the shared encoder E syn The encoding features of .

[0050] Taking the conversion of an image in the ma mode to an image in the corresponding mb mode as an example, Figure 1 In the above example, after the cross-modal generative adversarial learning network, the ma modal image is converted to the unregistered mb modal image. The real mb modal image and the generated image are sent to the discriminator to distinguish the authenticity of these images. The adversarial loss L ADV Constraints are imposed, and the adversarial learning network model continuously updates parameters to make the generated MB modal image close to the real image, trying to make the discriminator make wrong judgments. The discriminator also continuously optimizes parameters to minimize the probability of its wrong judgment. This adversarial process continues to make the generated image as close to the real MB modal image as possible.

[0051] At the same time, in order to maintain the integrity of geometry and boundaries in cross-modal image translation, the images of ma modality are translated through a shared encoder E in the adversarial learning network. syn and auxiliary segmentation decoder U seg , output the predicted label of ma mode , and the label of the source image of the ma modality A hybrid segmentation loss consisting of a combination of cross entropy loss and Dice loss is calculated together to achieve region-aware segmentation constraints. By contrastive learning, the focus is placed on modality-invariant structural features, and the source and target domain features are aligned from an independent perspective of modality-specific information. The process is as follows Figure 2 shown.

[0052] During the training process, in order to obtain the synthesized target-like image, this embodiment designs three different types of loss functions. The total loss of the translation network includes:

[0053] ,

[0054] Where, L ADV Indicates resistance loss; L SC represents the region-aware segmentation loss; L CL represents the contrast consistency loss; λ1, λ2, and λ3 are weight coefficients. The product of the weight coefficient and each loss reflects the current importance of the task in network training.

[0055] During the image translation training process, the importance of different tasks shows obvious stage differences over time. However, the traditional static weight allocation method lacks flexibility in this dynamic environment and cannot adaptively adjust to reflect the changes in task importance during the entire training process. In order to effectively balance the impact of different tasks at different stages, this embodiment adopts a dynamic weight control mechanism to adjust the weight coefficients of different loss functions.

[0056] First, for the weight coefficients of the three different loss functions, λ1, λ2, and λ3 need to be initialized as follows:

[0057] ,

[0058] In the formula, λ total Represents the sum of the weight values ​​of all task loss functions and is set to 1.

[0059] Subsequently, different strategies are used to adjust the weight coefficients for different loss functions.

[0060] For λ1, a dynamic adjustment mechanism based on the epoch number is introduced. The adjustment mechanism is as follows:

[0061] ,

[0062] Where f(e) is the nonlinear attenuation function; σ is a hyperparameter used to adjust the attenuation rate; N ep Indicates the number of training epochs.

[0063] For λ2 and λ3, a weight adjustment mechanism based on gradient descent is adopted, and the adjustment mechanism is as follows:

[0064] ,

[0065] ,

[0066] In the formula, Represents the loss function The gradient of the vector parameter θ; θ i represents the i-th parameter in the parameter vector θ.

[0067] The L2 gradient norm is used to evaluate the size of the task. If the L2 gradient norm is relatively large, it means that the task learns too fast relative to other tasks in the current iteration, so the weight of the task loss is reduced. On the contrary, if the L2 gradient norm is small, it means that the task learns slowly, so the weight of the task loss is increased.

[0068] In this embodiment, L SC Taking the dynamic weight control mechanism of λ2 as an example, the formula is as follows:

[0069] ,

[0070] ,

[0071] Where N=3, G avg represents the average gradient norm of all tasks.

[0072] Among them, the confrontation losses include:

[0073] ,

[0074] In the formula, and Denote the real image and the generated image respectively; D syn represents the discriminator.

[0075] In order to make the model focus more on the accuracy of image segmentation rather than the visual effect of the image, a region-aware segmentation loss is used to align the visual quality with the segmentation utility. In the decoding process of image translation, an auxiliary segmentation decoder U is deployed seg To explain the shared encoder E synThe encoded features are designed to minimize the region-aware segmentation loss and maintain the integrity of geometry and boundaries in cross-modal image translation.

[0076] Among them, the region-aware segmentation loss includes:

[0077] ,

[0078] Where, L Dice represents Dice loss; L CE represents the cross entropy loss; and denote the source image label and the predicted label respectively.

[0079] This embodiment also provides a contrast regularization for extracting styleless structural information. , }Apply three different random data augmentation transformations to generate { , Specifically, they are: (1) rotation prediction, which aims to capture the orientation and structural details of the image. (2) image denoising, which helps to extract deep local features and intrinsic features from noisy edge regions. (3) mask inpainting, which is used to identify the texture interaction between the occluded region and its adjacent context. Inspired by multi-view contrastive learning, { , }and{ , } is defined as positive. Accordingly, { , }、{ , }、{ , }、{ , } has different semantics but the same modality is defined as a negative pair, where i≠j and i,j∈N; where the subscript real represents the real sample image and the subscript syn represents the generated sample image. All enhanced data are input into the encoder of the segmentation network to produce a set of representative features. A discriminant function h is trained θ ( ), which is used to assign high values ​​to positive pairs and low values ​​to negative pairs. Then, instruct the function to select from the set G = {X, Y1, Y2, ..., Y k} accurately select a single positive sample X and k negative samples Y.

[0080] ,

[0081] Where, L con represents contrast loss; It means calculating the expected value in [ ] for the sample set G.

[0082] In line with the setting of contrastive learning, this embodiment establishes a sample set G={ , ,…… }, the above can be re-expressed as:

[0083] ,

[0084] ,

[0085] In the formula, Represents all possible sample sets { , ,…… }Calculate the expected value in [ ];

[0086] represents a fixed ma modal sample image; It represents samples with the same semantics obtained by converting a fixed ma modality image through an adversarial learning network to the mb modality; Express Apply three different random data augmentation transformations to obtain multiple samples with the same modality but different semantics; ( )and ( ) represent two encoders respectively; τ represents the temperature coefficient.

[0087] Therefore, the contrastive consistency loss consists of:

[0088] ,

[0089] In the formula, and represents the contrast loss, Represents a fixed ma modality image, by maximizing the positive sample pair and , while minimizing the similarity of positive and negative sample pairs ( , ) The difference in similarity helps the model learn semantically identical features in different modalities; Same reason.

[0090] Then, the segmentation model is trained. In the segmentation stage, the classic Unet network is used as the segmentation model. First, the aligned images are spliced ​​and then input into the Unet network. The prediction obtained is compared with the image label to calculate the loss. For the medical image segmentation task, this embodiment adopts a hybrid segmentation loss that combines the cross entropy loss and the Dice loss. The formula is as follows:

[0091] ,

[0092] Where p and y represent the predicted labels of the segmentation network and the true labels of the source images, respectively.

[0093] like Figure 3 As shown in the figure, taking the aligned ma modal image and the corresponding generated mb modal image as an example, the ma modal image and the corresponding generated mb modal image are input into the U-Net segmentation network, the predicted probability map of the ma modality is output, and the combined loss (cross entropy loss and Dice loss) of the binary ma modal label and each pixel in the predicted probability map is calculated to complete the segmentation.

[0094] By comparing with the baseline model (CycleGAN+Unet), the proposed model achieved better segmentation effect in the cardiac substructure segmentation task, showing significant performance advantages.

[0095] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multimodal medical image segmentation system, characterized in that: include: Image generation module and segmentation module; The image generation module is used to perform cross-modal image generation on the unpaired image to obtain a generated image; The generated image is subjected to contrast learning to obtain a registration image; The segmentation module is used to perform a segmentation operation on the input registration image and the source image using a segmentation model to obtain a segmented image; The image generation module uses a translation network to perform a cross-modal transformation operation on the unpaired image to obtain the generated image; The translation network includes two adversarial learning networks with the same structure; each adversarial learning network includes a shared encoder, a translation decoder, a discriminator and an auxiliary segmentation decoder; The source image passes through the shared encoder and the translation decoder to obtain the generated image of a different modality from the source image; Inputting the generated image and a real image of the same modality as the generated image into the discriminator, so that the generated image approaches the real image of the same modality as the generated image; The source image passes through the shared encoder and the auxiliary segmentation decoder, outputs the predicted label of the source image, and calculates the mixed segmentation loss with the predicted label and the true label of the source image to complete the segmentation constraint.

2. A multimodal medical image segmentation system according to claim 1, characterized in that: The total loss of the translation network includes: , In the formula, L ADV It means fighting against loss; L SC represents the region-aware segmentation loss; L CL represents contrast consistency loss; λ 1 , 2 and λ 3 is the weight coefficient.

3. A multimodal medical image segmentation system according to claim 2, characterized in that: A dynamic weight control mechanism is used to adjust the weight coefficient; for λ 1. The adjustment mechanism is as follows: , In the formula, f ( e ) is a nonlinear attenuation function; σ is a hyperparameter used to adjust the decay rate; N ep Indicates the number of training rounds epoch; for λ 2. The adjustment mechanism is as follows: , , ; for λ 3. The adjustment mechanism is as follows: , , ; In the formula, Represents the loss function For parameter vector θ The gradient of θ i Represents the parameter vector θ The i parameters; N =3, Represents the parameter vector θ The i parameters.

4. A multimodal medical image segmentation system according to claim 2, characterized in that: The adversarial losses include: , In the formula, and Represent the real image and the generated image respectively; D syn represents the discriminator.

5. A multimodal medical image segmentation system according to claim 2, characterized in that: The region-aware segmentation loss includes: , In the formula, L Dice means Dice loss; L CE stands for cross entropy loss; and denote the source image label and the predicted label respectively.

6. A multimodal medical image segmentation system according to claim 2, characterized in that: The comparison consistency loss includes: , In the formula, and represents contrast loss; in, , , In the formula, Represents all possible sample sets { , ,…… }Calculate the expected value in [ ]; represents a fixed ma modal sample image; It represents samples with the same semantics obtained by converting a fixed ma modality image through an adversarial learning network to the mb modality; ( j ≠1) is correct Apply three different random data augmentation transformations to obtain multiple samples with the same modality but different semantics; ( )and ( ) represent two encoders respectively; Represents the temperature coefficient.

7. A multimodal medical image segmentation system according to claim 1, characterized in that: The segmentation model adopts the Unet network; the losses of the Unet network include: , In the formula, L Dice means Dice loss; L CE stands for cross entropy loss; p , y denote the predicted labels of the segmentation network and the true labels of the source images, respectively.