A cross-modal image style transfer method, system and medium

Abstract

The application discloses a cross-modal image style migration method and system and a medium, which are used for converting an image of a first modal type into an image of a second modal type. The method comprises the following steps: acquiring an initial image of the first modal type of an object, and inputting the initial image into a pre-constructed cross-modal style migration model. The model is established by the following steps: inputting first and second modal type images in samples into first and second encoders respectively to obtain first and second image features correspondingly; training a basic model by using a preset total loss function, so that the trained model can align the first and second image features; the total loss function is at least related to a similarity loss, and the similarity loss is determined according to the first and second image features; after the training, the model extracts target image features from the initial image by using the first encoder, and reconstructs a target image of the second modal type based on the target image features. The application can improve the precision, clarity and accuracy of cross-modal image conversion.

Description

Technical Field

[0001] This invention relates to the fields of medical devices and computer vision technology, and in particular to a cross-modal image style transfer method, system and medium. Background Technology

[0002] In interventional therapy, for liver space-occupying lesions that lack typical imaging features of liver cancer, a clear pathological diagnosis can be expected through liver lesion puncture biopsy ultrasound technology or CT guidance. Although the updating of surgical imaging equipment and technology has improved the accuracy and efficiency of the puncture process, the low signal-to-noise ratio of intraoperative CT imaging increases the uncertainty in the puncture process.

[0003] Computed tomography (CT) imaging is a non-invasive imaging method that provides detailed cross-sectional images and is widely used in clinical observation, disease diagnosis, and treatment guidance. Contrast-enhanced CT significantly improves image contrast and clarity by injecting an iodine contrast agent into the patient, thereby increasing the visibility of specific tissues or blood vessels. This technology helps doctors more accurately identify lesions or other abnormalities. [1] However, due to limitations in contrast agent use and other factors during surgery, contrast-free plain CT scans are typically used as the primary imaging method. Therefore, image generation models based on convolutional neural networks have significant potential for achieving cross-modal medical image style transfer.

[0004] The emergence of Generative Adversarial Networks (GANs) and diffusion models has laid a strong theoretical foundation for image generation and opened up new possibilities for cross-modal style transfer in medical images. In recent years, medical applications based on these generative models have increased significantly, such as CycleGAN. [2] Pix2pix [3] DDPM [4] DDIM [5] These technologies have received widespread attention and application. Furthermore, to achieve lightweight models, the combination of GANs and diffusion models, such as LDM, has gradually come into the researchers' view. [6] ALDM [7] These generative models not only capture and reconstruct complex image features, but also optimize the quality of style transfer-generated images through sophisticated model design and theoretical derivation.

[0005] Currently, mainstream image generation models both domestically and internationally typically use Stable Diffusion. [6]This forms the basic framework. However, Stable Diffusion has certain limitations in the application of medical imaging. First, in medical image processing, images are often 3D (such as CT or MRI scans). However, Stable Diffusion essentially processes 2D image data, which presents challenges when directly applied to 3D images. Second, the generation of medical images requires not only image clarity but also accurate reconstruction of anatomical structures and lesion features, but Stable Diffusion struggles to meet these requirements without proper guidance.

[0006] The relevant prior art references [1] to [7] cited above are as follows:

[0007] [1]Hansen N J.Computed tomographic angiography of the abdominal aorta[J]. Radiologic Clinics, 2016, 54(1):35-54.

[0008] [2]Zhu JY, Park T, Isola P, et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C] / / Proceedings of the IEEEinternational conference on computer vision.2017:2223-2232.

[0009] [3]Isola P, Zhu JY, Zhou T, et al.Image-to-image translation with conditional adversarial networks[C] / / Proceedings of the IEEE conference oncomputer vision and pattern recognition.2017:1125-1134.

[0010] [4]Ho J,Jain A,Abbeel P.Denoising diffusion probabilistic models[J].Advances in neural information processing systems,2020,33:6840-6851.

[0011] [5]Song J,Meng C,Ermon S.Denoising diffusion implicit models[J].arXivpreprint arXiv:2010.02502,2020.

[0012] [6] Rombach R, Blattmann A, Lorenz D, et al. High-resolution images synthesis with latent diffusion models [C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2022:10684-10695.

[0013] [7]Kim J, Park H.Adaptive latent diffusion model for 3d medical image to image translation:Multi-modal magnetic resonance imaging study[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of ComputerVision.2024:7604-7613.

[0014] The above background information is provided only to assist in understanding the inventive concept and technical solution of this invention. It does not necessarily belong to the prior art of this patent application, nor does it necessarily provide technical teaching. In the absence of clear evidence that the above information was disclosed before the filing date of this patent application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0015] The purpose of this invention is to provide a cross-modal image style transfer method, system, and medium that can effectively reduce problems such as low accuracy, unclearness, and inaccuracy in cross-modal image conversion caused by pixel and anatomical misalignment.

[0016] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0017] A cross-modal image style transfer method for converting an image of a first modality type into an image of a second modality type, the method comprising the following steps:

[0018] Obtain an initial image of an object in its first modality type, and input the initial image into a pre-built cross-modal style transfer model;

[0019] The cross-modal style transfer model is established in advance through the following steps:

[0020] Collect a learning sample set, wherein each sample in the learning sample set includes a first modality type image and a second modality type image of the same object;

[0021] Design a basic model, the basic model including a first encoder and a second encoder; wherein the first encoder and the second encoder are configured to extract features from an image;

[0022] Using the aforementioned training sample set, the base model is optimized and trained in the following manner:

[0023] The first modality type image in the sample is input into the first encoder to obtain the first image feature; and the second modality type image in the sample is input into the second encoder to obtain the second image feature;

[0024] The base model is trained using a preset total loss function, so that the first image features extracted by the first encoder of the trained model are aligned with the second image features extracted by the second encoder, thus obtaining the cross-modal style transfer model; the total loss function is at least related to the similarity loss, which is determined based on the first image features and the second image features;

[0025] The cross-modal style transfer model uses a first encoder to extract target image features from the initial image, and reconstructs a target image of the second modality based on the target image features.

[0026] Furthermore, following any one or a combination of the aforementioned technical solutions, the basic model further includes a quantization module. During the optimization training process of the basic model, the quantization module is configured to quantize the first image features to obtain quantized features. The total loss function is also at least related to the quantization loss, which is determined based on the first image features and the quantized features.

[0027] The cross-modal style transfer model uses its quantization module to quantize the target image features to obtain target quantized features, and reconstructs the target image of the second modality based on the target quantized features.

[0028] Furthermore, based on any one or a combination of the aforementioned technical solutions, the formula for calculating the quantization loss is expressed as follows:

[0029]

[0030] Among them, L quan E represents the quantified loss. c This represents the second image feature. `detach()` indicates pausing gradient backpropagation during model training. This represents the quantization feature.

[0031] Furthermore, following any one or a combination of the aforementioned technical solutions, the base model further includes a decoder. During the optimization training process of the base model, the decoder is configured to reconstruct an image of the second modality type based on the second image features to obtain a reconstructed image of the second modality type. The total loss function is also at least related to the reconstruction loss, which is based on the image of the second modality type and the reconstructed image of the second modality type.

[0032] The cross-modal style transfer model uses its decoder to reconstruct a target image of the second modality type based on the features of the target image.

[0033] Furthermore, based on any one or a combination of the aforementioned technical solutions, the formula for calculating the reconstruction loss is expressed as follows:

[0034]

[0035] Among them, L rec Let x represent the reconstruction loss. c This represents the second modality type image. This indicates that the image is reconstructed using the second modality type.

[0036] Furthermore, following any one or a combination of the aforementioned technical solutions, the base model further includes a classifier. During the optimization training of the base model, the classifier is configured to determine a classification loss, which is determined based on the second image and a reconstructed image of the second modality type obtained based on the features of the second image. The total loss function is also at least related to the classification loss.

[0037] Furthermore, based on any one or a combination of the aforementioned technical solutions, the formula for calculating the classification loss is expressed as follows:

[0038]

[0039] Among them, L dis Let x represent the classification loss, D(·) represent the classifier, and x represent the classification loss. c This represents the second modality type image. This indicates that the image is reconstructed using the second modality type.

[0040] Furthermore, following any one or a combination of the aforementioned technical solutions, the similarity loss is determined based on the first image feature and the second image feature, and the formula for calculating the similarity loss is expressed as follows:

[0041] L sim =||E c -E n ||2

[0042] Among them, L sim E represents the similarity loss. c E represents the second image feature. n This represents the first image feature.

[0043] Furthermore, in accordance with any or a combination of the aforementioned technical solutions, the basic model further includes a feature optimizer, which is configured to optimize the features extracted from the image by the first encoder.

[0044] The cross-modal style transfer model utilizes the feature optimizer therein to optimize the target image features to obtain optimized features, and reconstructs the target image of the second modality based on the optimized features.

[0045] Furthermore, following any one or a combination of the aforementioned technical solutions, the feature optimizer includes a 3D convolutional layer, a normalization layer, and an activation function layer connected in sequence.

[0046] Furthermore, following any or a combination of the aforementioned technical solutions, the method further includes converting the target image features into target fusion features using a pre-constructed feature space diffusion model; the cross-modal style transfer model reconstructs a target image of a second modality type based on the target fusion features;

[0047] The feature space diffusion model is established in advance through the following steps:

[0048] Collect a diffusion learning sample set, wherein each sample in the diffusion learning sample set includes the first image feature and the second image feature;

[0049] Design a basic diffusion model, which includes a dynamic similarity mask module and a diffusion module. The dynamic similarity mask module is configured to obtain a mask for its input image features.

[0050] Using the aforementioned diffusion learning sample set, the basic diffusion model is optimized and trained in the following manner:

[0051] The first image feature and the second image feature are input into the dynamic similarity mask module to obtain a dynamic similarity mask for the first image feature and the second image feature;

[0052] The first image feature and the dynamic similarity mask are input into the diffusion module to obtain the first fused feature;

[0053] The basic diffusion model is trained using a preset diffusion loss function, so that the first fusion feature output by the trained model incorporates the structural information of the second image feature on the basis of the first image feature, thus obtaining the feature space diffusion model; the preset diffusion loss function is at least related to the dynamic similarity mask;

[0054] The feature space diffusion model utilizes its diffusion module to incorporate the structural information of the image features of the second modality type image into the target image features to obtain the first target fusion feature, and determines the target fusion feature based on the first target fusion feature.

[0055] Furthermore, following any one or a combination of the aforementioned technical solutions, the basic diffusion model further includes a noise processing module, which is configured to add noise and / or denoise the first fusion feature to obtain a second fusion feature; the diffusion loss function is also related to at least the input noise and output noise of the basic diffusion model, wherein the input noise is determined by the first image feature and the second image feature, and the output noise is determined by the second fusion feature;

[0056] The feature space diffusion model uses the noise processing module therein to add noise and / or denoise the first target fusion feature to obtain the second target fusion feature; and determines the second target fusion feature as the target fusion feature.

[0057] Furthermore, following any one or a combination of the aforementioned technical solutions, the diffusion loss function is expressed by the following formula:

[0058]

[0059] Wherein, DSM represents the dynamic similarity mask, E n E represents the first image feature. c This represents the second image feature. This represents the dot product operation. This represents the random noise ∈ denoted by E, sampled from the standard normal distribution (N(0,1)) at each time step t, combined with the input data E. n and E cPerform multiple samplings, output noise ∈ θ The expected error between the input noise (i.e., the true noise) and the input noise (i.e., the true noise), where ∈ represents the input noise. θ This refers to the output noise. This represents the square of the L2 norm.

[0060] Furthermore, based on any one or a combination of the aforementioned technical solutions, the calculation formula for the dynamic similarity mask is as follows:

[0061]

[0062] Where DSM represents the dynamic similarity mask, τ represents the current training epoch number, Total epochs represents the total number of training epochs, and <·,·> represents the cosine similarity between the first image feature and the second image feature, with the value range of <·,·> being [-1, 1]. The similarity range is changed to [0,1], min[·,·] means taking the minimum value between two elements, and α is a scaling factor, which is a preset value.

[0063] Furthermore, following any one or a combination of the aforementioned technical solutions, the method further includes: determining the first target fusion feature as the target fusion feature.

[0064] According to another aspect of the present invention, the present invention provides a cross-modal image style transfer system, which uses the cross-modal image style transfer method described above, or a combination of multiple technical solutions, to convert an image of a first modality type into an image of a second modality type.

[0065] According to another aspect of the present invention, a computer-readable storage medium is provided for storing program instructions configured to be invoked to perform the steps of the method described in any one of the above technical solutions or a combination of multiple technical solutions.

[0066] The beneficial effects of the technical solution provided by this invention are as follows:

[0067] a. The cross-modal image style transfer method provided by this invention achieves feature-level alignment of cross-modal medical images through representation learning during the process of converting images of a first modality type to images of a second modality type. This effectively reduces the problems caused by pixel and anatomical structure misalignment in cross-modal data in practical applications. Specifically, the cross-modal image style transfer method cleverly decomposes the style transfer task from a first modality type image to a second modality type image into two sub-tasks: first, to achieve feature-level alignment between the first modality type image and the second modality type image; and second, to complete the reconstruction of the second modality type image itself. Since the input and output data are both second modality type images during the reconstruction process in the training process, the pixel mismatch problem is avoided. The L2 norm is used to supervise and constrain the features of the first modality type image and the second modality type image, thereby indirectly achieving style transfer from the first modality type image to the second modality type image. In this process, a feature distribution with semantic consistency is obtained, thereby enabling high-quality conversion of a first modality type image into a first modality type image with a different modality type.

[0068] b. The cross-modal medical image style transfer method based on the feature space diffusion model of this invention can adjust the operation of the feature space diffusion model in the noise addition and denoising process based on a dynamic similarity mask. It reveals the characteristics of the feature distribution corresponding to the two modalities of the image. Taking plain CT images and contrast-enhanced CT images as examples, similar regions correspond to anatomical structures that are similar in the two CT data, while dissimilar regions correspond to areas in the contrast-enhanced CT image that differ from the plain CT image due to the addition of contrast agent. As the model training iterates, the focus of the feature space diffusion model dynamically shifts from semantically rich large structural similar regions to contrast-enhanced regions with obvious contrast, thereby further improving the generation performance and efficiency of this invention. Furthermore, compared with hard conditions such as segmentation results and text prompts, the similarity mask obtained through the model's posterior is easier to obtain in practical applications and can be more directly and effectively integrated into the model training process. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 A schematic diagram of the architecture of a first cross-modal style transfer model provided as an exemplary embodiment of the present invention;

[0071] Figure 2A schematic diagram illustrating the training process of a first cross-modal style transfer model provided as an exemplary embodiment of the present invention;

[0072] Figure 3 A schematic diagram illustrating the application process of a first cross-modal style transfer model provided as an exemplary embodiment of the present invention;

[0073] Figure 4 A flowchart illustrating a first cross-modal image style transfer method provided as an exemplary embodiment of the present invention;

[0074] Figure 5 A schematic diagram of the architecture of a feature space diffusion model provided as an exemplary embodiment of the present invention;

[0075] Figure 6 A schematic diagram illustrating the training process of a feature space diffusion model with features as input, provided as an exemplary embodiment of the present invention;

[0076] Figure 7 A schematic diagram illustrating the application process of a feature space diffusion model provided as an exemplary embodiment of the present invention;

[0077] Figure 8 A schematic diagram illustrating the training process of a feature space diffusion model with an image as input, provided as an exemplary embodiment of the present invention;

[0078] Figure 9 A schematic diagram illustrating the training process of a pixel space reconstruction model provided as an exemplary embodiment of the present invention;

[0079] Figure 10 This is a schematic diagram illustrating the application process of a second cross-modal style transfer model based on a feature space diffusion model, provided as an exemplary embodiment of the present invention. Detailed Implementation

[0080] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0081] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0082] Based on the existing technology, this application finds that the existing technology has at least the following shortcomings:

[0083] I. Insufficient Accuracy of Pixel Alignment in Cross-Modal Images: Due to patient respiratory movements and changes in various tissues or organs at different time points, the pixel space registration between plain CT and contrast-enhanced CT scans is often less than ideal. Mainstream generative models (such as Pix2pix) typically assume that the input and output images are accurately aligned and rely on this premise to learn the complex mapping relationships between paired images. However, this assumption may limit the model's generalization ability in practical applications. [8] This limitation is particularly evident in the field of medical imaging, where the precision of anatomical structures is required to be even higher.

[0084] II. Guidance in feature space cannot simultaneously achieve both image generation accuracy and focus on the target region: Current image generation models can be broadly categorized into conditional guidance. [9] and unconditional guidance

[10] Two technical approaches. Their main difference lies in whether conditional information is introduced into the model training process through an additional encoding mechanism. Conditionally guided models usually enhance the model's learning of these specific regions or content by encoding specific conditions (such as the segmentation results of regions of interest in an image or text descriptions) into feature vectors and fusing them with the original image features. This approach enables the model to focus more on key regions, thereby more accurately representing the target features in the generated results. Although unconditionally guided models can operate without relying on certain specific conditions, they lack accuracy in generating specific regions due to the lack of targeted guidance. In practice, the generation process often lacks the supplementation of specific conditions, making it difficult for the model to focus on the anatomical structures or lesion regions of interest. The references cited above [8] to

[10] are as follows:

[0085] [8]Kong L, Lian C, Huang D, et al. Breaking the dilemma of medical image-to-image translation[J]. Advances in Neural Information Processing Systems, 2021,34:1964-1978.

[0086] [9]Li Y, Shao HC, Liang X, et al. Zero-shot medical image translation via frequency-guided diffusion models [J]. IEEE transactions on medical imaging, 2023.

[0087]

[10] M,Dalmaz O,Dar SUH,et al.Unsupervised medical image translation with adversarial diffusion models[J].IEEE Transactions on MedicalImaging,2023.

[0088] Based on the shortcomings of existing technologies mentioned above, and combined with the analysis of practical applications and model accuracy, this invention focuses on reducing the model error range caused by unaligned paired data and optimizing the unconditional guidance process. Therefore, based on existing mainstream models, this invention proposes an unconditional guidance model suitable for cross-modal generation of medical images, which can better meet the actual needs of the medical image generation process.

[0089] In one embodiment of the present invention, a cross-modal image style transfer method is provided for converting an image of a first modality type into an image of a second modality type. See [link to relevant documentation]. Figure 1 and Figure 2 The method includes the following steps:

[0090] Obtain an initial image of an object in its first modality type, and input the initial image into a pre-built cross-modal style transfer model;

[0091] The cross-modal style transfer model is established in advance through the following steps:

[0092] Collect a learning sample set, wherein each sample in the learning sample set includes a first modality type image and a second modality type image of the same object;

[0093] Design a basic model, the basic model including a first encoder and a second encoder; wherein the first encoder and the second encoder are configured to extract features from an image;

[0094] Using the aforementioned training sample set, the base model is optimized and trained in the following manner:

[0095] The first modality type image in the sample is input into the first encoder to obtain the first image feature; and the second modality type image in the sample is input into the second encoder to obtain the second image feature;

[0096] The base model is trained using a preset total loss function, so that the first image features extracted by the first encoder of the trained model are aligned with the second image features extracted by the second encoder, thus obtaining the cross-modal style transfer model; the total loss function is at least related to the similarity loss, which is determined based on the first image features and the second image features;

[0097] The cross-modal style transfer model uses a first encoder to extract target image features from the initial image, and reconstructs a target image of the second modality based on the target image features.

[0098] Following clinical guidelines, clinicians often prefer to use plain CT scans over contrast-enhanced CT scans. However, plain CT scans often lack sufficient clarity in visualizing anatomical structures. To address this issue, many generative models are trained one-to-one on paired plain and contrast-enhanced CT data to improve the clarity of anatomical structures on plain CT scans. However, most studies suffer from model accuracy issues due to image misalignment and insufficient utilization of cross-modal data.

[0099] Therefore, based on the theoretical foundation of representation learning, this invention designs a dual-encoder generative structure and multiple combined loss functions to obtain superior feature representations. For example... Figure 1 As shown, for paired plain CT scans (x... n ) and contrast-enhanced CT (x-ray) c This invention obtains the features of both CT scans simultaneously using dual encoders, namely the features of plain CT scans (E). n ) and the characteristics of the enhancement period (E) c In this process, the dual encoder consists of two encoders with identical structures, and the similarity loss (L...) is used to... sim This is achieved by aligning paired input images in the feature space. Notably, to achieve better alignment, a feature optimizer is added to the encoder of the plain CT input to obtain a better representation of the plain CT image features (P).n ).

[0100] In one embodiment of the present invention, the base model further includes a feature optimizer, a quantization module, a decoder, and a classifier. The feature optimizer is configured to optimize its input features to obtain optimized features. Specifically, the feature optimizer mainly consists of a 3D convolutional layer (Conv3d), a normalization layer (Group Norm), and an activation function layer (Sigmoid) connected in sequence, which performs deeper feature extraction on the input features to obtain information that better reflects their structural characteristics.

[0101] The quantization module is configured to quantize its input features to obtain quantized features. The decoder is configured to decode its input image features to obtain a reconstructed image; the classifier is configured to penalize regions with poor reconstruction quality in the reconstructed image to improve the quality of the reconstructed image. Generally, adversarial generative networks mainly consist of a generator and a classifier, which achieve high-quality image generation through adversarial interaction. This invention, based on the structure of adversarial generative networks, also designs a classifier (Discriminator) to discriminate the reconstructed image, using a classification loss (L... dis This penalizes areas with poor reconstruction results, thereby improving the quality of the reconstructed image.

[0102] In one embodiment of the present invention, such as Figure 1 , Figure 2 and Figure 4 As shown, the process of optimizing and training the base model also includes the following steps.

[0103] The first image features are quantized using the quantization module to obtain quantized features; the total loss function is also at least related to the quantization loss, which is determined based on the first image features and the quantized features.

[0104] Regarding the quantization loss, preferably, the square of the L2 norm is applied to both the second image feature and the quantized feature. The weighted calculation is performed, and the formula for calculating the quantization loss is expressed as follows:

[0105]

[0106] Among them, L quan E represents the quantified loss. c This represents the second image feature. `detach()` indicates pausing gradient backpropagation during model training. This represents the quantization feature.

[0107] The decoder is used to reconstruct an image of the second modality type based on the second image features to obtain a reconstructed image of the second modality type; the total loss function is also at least related to the reconstruction loss, which is based on the second modality type image and the reconstructed image of the second modality type.

[0108] Regarding the reconstruction loss L rec Preferably, the enhancement-phase CT (x) is calculated based on the square of the L1 norm. c ) and the enhanced CT after reconstruction The degree of similarity between the two is expressed by the following formula for calculating the reconstruction loss:

[0109]

[0110] Among them, L rec Let x represent the reconstruction loss. c This represents the second modality type image. This indicates that the image is reconstructed using the second modality type.

[0111] The classifier is configured to determine a classification loss based on the second image and a reconstructed image of a second modality type obtained from the features of the second image; the total loss function is also at least related to the classification loss.

[0112] Regarding classification loss L dis Preferably, the classification loss is calculated using a binary cross-entropy loss function for both the second modality image and the reconstructed second modality image. The formula for calculating the classification loss is as follows:

[0113]

[0114] Among them, L dis Let x represent the classification loss, D(·) represent the classifier, and x represent the classification loss. c This represents the second modality type image. This indicates that the image is reconstructed using the second modality type.

[0115] In this embodiment, during the process of determining the similarity loss based on the first image features and the second image features, the feature optimizer is first used to optimize the first image features extracted from the image by the first encoder to obtain optimized features. Then, the similarity loss is calculated based on the optimized features and the second image features.

[0116] Regarding similarity loss L sim Preferably, the optimized feature (P) is optimized using the L2 norm. n ) and second image features (E c The calculation is performed using the formula shown below.

[0117] L sim =||E c -P n ||2

[0118] Among them, L sim E represents the similarity loss. c P represents the second image feature. n This refers to the optimized feature.

[0119] All the loss functions involved in the above parts participate in the training process of the autoencoder. That is, during the training process, the quantization loss, reconstruction loss, classification loss and similarity loss must be less than their respective preset thresholds so that their corresponding modules meet the training requirements.

[0120] Regarding the unified paradigm for various losses, this invention designs the following total loss function based on VQ-GAN, and the calculation method of the total loss function is expressed as follows:

[0121] L total =α r L rec +α q L quan +α s L sim +α c L dis

[0122] Where, α r α q α s α d These represent the preset weight values ​​for reconstruction loss, quantization loss, similarity loss, and classification loss, respectively.

[0123] By designing the loss function of the above combination, the autoencoder can not only generate high-quality images, but also effectively align features of cross-modal data (plain CT and enhanced CT). This feature alignment provides optimization space for denoising based on the diffusion model in the feature space, thereby further improving the generation performance and application effect of this invention in converting images of the first modality type into images of the second modality type.

[0124] In this embodiment, model training is completed when the total loss function is less than a preset threshold, and the trained model is the cross-modal style transfer model. The steps for converting an image of the first modality type into an image of the second modality type using the cross-modal style transfer model are as follows: Figure 3 As shown.

[0125] An initial image of an object in a first modality is obtained and input into a pre-constructed cross-modal style transfer model. The cross-modal style transfer model uses a first encoder to extract target image features from the initial image and outputs the feature optimizer. The feature optimizer optimizes the target image features to obtain optimized features and outputs them to the quantization module. The quantization module quantizes the target image features to obtain target quantized features and outputs them to an adversarial generative network module composed of a decoder and a classifier. The decoder decodes and reconstructs the target quantized features to obtain a target image in a second modality.

[0126] In another embodiment of the present invention, the design of the total loss function differs from that of the above embodiments. In this embodiment, the total loss function is not determined based on the quantization loss, reconstruction loss, classification loss, and similarity loss, but rather based on a portion of the quantization loss, reconstruction loss, classification loss, and similarity loss.

[0127] In another embodiment of the present invention, the difference from the above embodiment lies in the feature optimizer. In this embodiment, the feature optimizer is not configured at the output of the first encoder, regarding the similarity loss L. sim It is calculated directly based on the first image feature and the image feature, and its calculation formula is expressed as follows:

[0128] L sim =||E c -E n ||2

[0129] Among them, L sim E represents the similarity loss. c E represents the second image feature. n This represents the first image feature.

[0130] The cross-modal image style transfer method provided in the above embodiments achieves feature-level alignment of cross-modal medical images through representation learning during the process of converting images of the first modality type into images of the second modality type. This effectively reduces the problems caused by pixel and anatomical structure misalignment in cross-modal data in practical applications.

[0131] In practical applications, taking plain CT images and enhanced CT images as examples, i.e., the plain CT image is the first image and the enhanced CT image is the second image, the above-mentioned cross-modal image style transfer method cleverly decomposes the style transfer task from plain CT to enhanced CT into two sub-tasks: one is to achieve alignment between plain CT and enhanced CT at the feature level ( Figure 2 The process is illustrated by the two dotted lines at the bottom center; secondly, it involves completing the reconstruction of the enhanced CT scan itself. Figure 2 (The process is illustrated by the upper half-underline). Since both input and output data are enhanced CT scans during reconstruction, pixel mismatch issues are avoided. Simultaneously, by using the L2 norm to supervise and constrain the features of plain and enhanced CT scans, style transfer from plain to enhanced CT is indirectly achieved. In this process, a semantically consistent feature distribution is obtained, thus enabling high-quality conversion of plain CT images into enhanced CT scans with different modalities.

[0132] The cross-modal image style transfer method described above is a technical solution proposed in this invention based on pixel space to convert images of a first modality type into images of a second modality type. Through model structure design and loss function constraints, feature alignment of cross-modal data is achieved. However, using an adversarial generative model as the generative model has certain instability in practical applications and also requires significant computational resources. Therefore, this invention introduces a diffusion model in the latent space, optimizing the feature distribution (L...) through maximum likelihood optimization. diff The calculation formula is used to construct a cross-modal style transfer model with more stable performance and higher computational efficiency.

[0133] In one embodiment of the present invention, a cross-modal image style transfer method is provided, see [link to relevant documentation]. Figures 5 to 7 The cross-modal image style transfer method further includes the following steps:

[0134] The target image features are converted into target fusion features using a pre-constructed feature space diffusion model; the cross-modal style transfer model reconstructs a target image of a second modality based on the target fusion features;

[0135] The feature space diffusion model is established in advance through the following steps:

[0136] Collect a diffusion learning sample set, wherein each sample in the diffusion learning sample set includes the first image feature and the second image feature;

[0137] Design a basic diffusion model, which includes a dynamic similarity mask module and a diffusion module. The dynamic similarity mask module is configured to obtain a mask of its input image features, and the diffusion module is configured to use the mask to perform a masking operation on the first image features to obtain fused features.

[0138] Using the aforementioned diffusion learning sample set, the basic diffusion model is optimized and trained in the following manner:

[0139] The first image feature and the second image feature are input into the dynamic similarity mask module to obtain a dynamic similarity mask for the first image feature and the second image feature;

[0140] The first image feature and the dynamic similarity mask are input into the diffusion module to obtain the first fused feature;

[0141] The basic diffusion model is trained using a preset diffusion loss function, so that the first fusion feature output by the trained model incorporates the structural information of the second image feature on the basis of the first image feature, thus obtaining the feature space diffusion model; the preset diffusion loss function is at least related to the dynamic similarity mask;

[0142] The feature space diffusion model utilizes its diffusion module to incorporate the relevant structural information of the image features of the second modality type image into the target image features to obtain the first target fusion feature, and determines the target fusion feature based on the first target fusion feature.

[0143] In one embodiment of the present invention, the first target fusion feature is determined to be the target fusion feature. In other preferred embodiments of the present invention, a noise addition process and a noise reduction process are introduced on the first target fusion feature to obtain a target fusion feature of better quality.

[0144] In one embodiment of the present invention, the basic diffusion model further includes a noise processing module. For example... Figure 5 As shown, the noise processing module includes a noise-adding module and a noise-reducing module, wherein the noise-adding module is configured to add noise to its input; and the noise-reducing module is configured to reduce noise from its input.

[0145] This invention calculates the similarity matrix between a first image feature (preferably an optimized feature) and a second image feature using cosine similarity, and uses this matrix as supplementary information to strengthen the original input feature of the model, namely the first image feature E. n Or the optimized feature P n To better utilize semantic information during noise addition and denoising, and to guide the model to actively learn sparsely distributed and low-similarity regions in plain and enhanced CT scans, this invention achieves this goal through changes in a Dynamic Similarity Mask (DSM). The calculation formula for the Dynamic Similarity Mask (DSM) is as follows:

[0146]

[0147] Where DSM represents the dynamic similarity mask, τ represents the current training epoch number, Total epochs represents the total number of training epochs, and <·,·> represents the cosine similarity between the first image feature and the second image feature, with the value range of <·,·> being [-1, 1]. This is the cosine similarity matrix between the first image feature and the second image feature, used to change the similarity range to [0,1]. min[·,·] represents taking the minimum value between two elements. In this formula, it is obtained by comparison. The minimum value is calculated by combining the magnitudes of 1 and α, which is a scaling factor and a preset value. α is used to control the rate of change of the period ratio.

[0148] During the training process of the basic diffusion model, this invention designs a diffusion loss function (L... diff Combining dynamic similarity masking and denoising processes, the diffusion loss function is calculated as follows:

[0149]

[0150] Wherein, DSM represents the dynamic similarity mask, E n Let E represent the first image feature and E represent the second image feature. This represents the dot product operation. This represents the random noise ∈ denoted by E, sampled from the standard normal distribution (N(0,1)) at each time step t, combined with the input feature E. n and E c The output noise obtained by sampling multiple times and calculating the model ∈ θ The expected error between the input noise ∈ and the input noise ∈, where ∈ represents the input noise, which is determined by the first image feature and the second image feature, ∈ θ The output noise is represented by the second fusion feature; This indicates that the constraint is applied by the square of the L2 norm. The time step t is a value that is uniformly and randomly drawn from the set {1,…,T}, and T represents the total time steps in the denoising process.

[0151] The aforementioned diffusion loss function and the basic diffusion model are used for optimization training to obtain a trained model, which is then used as the feature space diffusion model. The target image features or target optimized features are used as inputs to the feature space diffusion model. At this point, one input to the dynamic similarity mask module is the target image features or target optimized features. Since there is no input item corresponding to the second image features during training, the other input is 0, and the dynamic similarity mask module is a matrix of all 1s. Based on this, the trained diffusion module adds image features of the second modality type to the target image features or target optimized features to obtain a first target fusion feature. The first target fusion feature is then further denoised to obtain a second fusion feature with added Gaussian noise. Finally, the second fusion feature undergoes a stepwise denoising process with a total of T steps to obtain a third fusion feature, which is used as the desired target fusion feature.

[0152] By introducing the feature space diffusion model into the cross-modal style transfer model, and using the target fusion feature as the input of the quantization module, the target fusion feature is sequentially quantized, decoded, and reconstructed to obtain the target image of the second modality.

[0153] In one embodiment of the present invention, a cross-modal medical image style transfer method is provided. The method is used to integrate distinguishing structural information possessed by a second modality image but not possessed by the first modality image into an image of a first modality type. (See [link to previous document]). Figure 8 In this embodiment, the method includes the following steps:

[0154] Obtain an initial image of the first modality type of an object, and input the initial image into a pre-constructed feature space diffusion model;

[0155] The feature space diffusion model is established in advance through the following steps:

[0156] Collect a learning sample set, wherein each sample in the learning sample set includes a first modality type image and a second modality type image of the same object;

[0157] Design a basic diffusion model, which includes an encoder module, a dynamic similarity mask module, and a diffusion module; wherein the encoder module is configured to extract features from an image;

[0158] Using the aforementioned training sample set, the base model is optimized and trained in the following manner:

[0159] The first modality type image from the sample is input into the encoder module to obtain the first image feature; and the second modality type image from the sample is input into the encoder module to obtain the second image feature;

[0160] The first image feature and the second image feature are input into the dynamic similarity mask module to obtain a dynamic similarity mask for the first image feature and the second image feature;

[0161] The first image feature and the dynamic similarity mask are input into the diffusion module to obtain the first fused feature;

[0162] The basic diffusion model is trained using a preset diffusion loss function, so that the first fusion feature output by the trained model incorporates the structural information of the second image feature that is different from that of the first image feature, on the basis of the first image feature, to obtain the feature space diffusion model; the diffusion loss function is at least related to the dynamic similarity mask;

[0163] The feature space diffusion model uses the encoder module to extract features from the initial image to obtain target image features, and uses the diffusion module to incorporate the structural information of the image features of the second modality type image into the target image features to obtain the first target fusion feature.

[0164] Preferably, the basic diffusion model further includes a noise processing module. For example... Figure 8 As shown, the noise processing module is configured to add noise and / or denoise the first fused feature to obtain the second fused feature; the diffusion loss function is also related to at least the input noise and output noise of the basic diffusion model, the input noise being determined by the first image feature and the second image feature, and the output noise being determined by the second fused feature;

[0165] The feature space diffusion model uses the noise processing module therein to add noise and / or denoise the first target fusion feature to obtain the second target fusion feature; and determines the second target fusion feature as the target fusion feature.

[0166] Preferably, the basic diffusion model further includes a feature optimizer configured to optimize its input image features to obtain optimized features, which serve as input to the noise processing module.

[0167] Wherein, the diffusion loss L diff The calculation formula is as follows:

[0168]

[0169] Wherein, DSM represents the dynamic similarity mask, and its calculation method is as described in the above embodiments, and will not be repeated here. E n This represents the first image feature. This represents the dot product operation. This represents the random noise ∈ denoted by x, sampled from a standard normal distribution (N(0,1)) at each time step t, combined with the input data x. n and x c Perform multiple samplings and calculate the noise in the model output ∈ θ The expected error between the input noise and the true noise ∈, where ∈ represents the input noise, ∈ θ This refers to the output noise. This indicates that constraints are applied using the square of the L2 norm, P. n For the optimized feature, the time step t is a value that is uniformly and randomly drawn from the set {1,…,T}, and T represents the total time steps in the denoising process.

[0170] The encoder module can be configured in two ways. In one configuration, the encoder module is a dual-encoder structure, comprising a first encoder and a second encoder. The first encoder is configured to extract features from a first modality image to obtain first image features, and the second encoder is configured to extract features from a second modality image to obtain second image features. In another configuration, the encoder module is a single-encoder structure, comprising only one encoder. The encoder extracts features from both the first and second modality images, and the first and second image features are distinguished by labels configured for each image.

[0171] In this embodiment, the cross-modal medical image style transfer method further includes converting an image of the first modality type into an image of the second modality type using a pre-established pixel space reconstruction model.

[0172] Preferably, see Figure 9 The basic model of the pixel space reconstruction model in this embodiment includes a feature encoder module, a feature optimizer, a quantization module, a decoder, and a classifier connected in sequence. In this embodiment, the functions of the feature optimizer, noise processing module, quantization module, decoder, and classifier are the same as those described in the previous embodiments, and will not be repeated here.

[0173] Using images of the first modality type and / or the second modality type as the learning sample set, the pixel space reconstruction model's base model is optimized and trained using the pixel space reconstruction loss function to obtain the trained model as the pixel space reconstruction model. In this embodiment, the pixel space reconstruction loss function is calculated as follows:

[0174] L′ total =k r L rec +k q L quan +k d L dis

[0175] Among them, L′ total L represents the pixel space reconstruction loss function. rec L represents the reconstruction loss. quan L represents the quantized loss. dis Let k represent the classification loss. r k q k d These represent the preset weight values ​​for reconstruction loss, quantization loss, diffusion loss, and classification loss, respectively.

[0176] It should be noted that the calculation methods for reconstruction loss, quantization loss and classification loss can be adopted using the calculation methods described in the above embodiments, and will not be repeated here.

[0177] See Figure 10 The trained diffusion model is added to the trained pixel space reconstruction model. Specifically, the feature encoder module and feature optimizer in the diffusion model and the pixel space reconstruction model are merged. The output of the diffusion model is used as the input of the quantization module in the pixel space reconstruction model. The quantization module and decoder in the pixel space reconstruction model quantize and reconstruct the output of the diffusion model (first target fusion feature / second target fusion feature) to obtain the target image of the second modality.

[0178] This invention provides a cross-modal medical image style transfer method based on a feature space diffusion model. This method adjusts the operation of the feature space diffusion model during noise addition and denoising based on a dynamic similarity mask, revealing the characteristics of the feature distribution corresponding to the two modalities of the image. Taking plain CT images and contrast-enhanced CT images as examples, similar regions correspond to anatomical structures that are similar in the two CT data, while dissimilar regions correspond to areas in the contrast-enhanced CT image that differ from the plain CT image due to the addition of contrast agent. As the model training iterates, the focus of the feature space diffusion model dynamically shifts from semantically rich large structural similar regions to contrast-enhanced regions with significant contrast, thereby further improving the generation performance and efficiency of this invention. Furthermore, compared to rigid conditions such as segmentation results and text prompts, the similarity mask obtained through the model's posterior is easier to obtain in practical applications and can be more directly and effectively integrated into the model training process.

[0179] In one embodiment of the present invention, a cross-modal image style transfer system is provided, which uses the cross-modal image style transfer method described in any of the above embodiments to convert an image of a first modality type into an image of a second modality type.

[0180] In one embodiment of the present invention, a computer-readable storage medium is provided for storing program instructions configured to be invoked to perform the steps of the method as described in any of the preceding embodiments.

[0181] It should be noted that the above-described cross-modal image style transfer system, computer-readable storage medium embodiments and cross-modal image style transfer method embodiments belong to the same inventive concept. The entire contents of the cross-modal image style transfer method embodiments are incorporated into the cross-modal image style transfer system, computer-readable storage medium embodiments by reference.

[0182] It should be noted that, in this document, relational terms are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0183] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A cross-modal image style transfer method for converting an image of a first modality type into an image of a second modality type, characterized in that, The method includes the following steps: Obtain an initial image of an object in its first modality type, and input the initial image into a pre-built cross-modal style transfer model; The cross-modal style transfer model is established in advance through the following steps: Collect a learning sample set, wherein each sample in the learning sample set includes a first modality type image and a second modality type image of the same object; Design a basic model comprising a first encoder, a feature optimizer, a second encoder, a quantization module, a classifier, and a decoder; wherein the first encoder and the second encoder are configured to extract features from an image; Using the aforementioned training sample set, the base model is optimized and trained in the following manner: The first modality type image from the sample is input into the first encoder to obtain the first image features. The feature optimizer optimizes the first image features to obtain optimized features. The feature optimizer includes a 3D convolutional layer, a normalization layer, and an activation function layer connected in sequence; and inputs the second modality type image from the sample into the second encoder to obtain the second image features. The quantization module is configured to quantize the second image features to obtain quantized features; the decoder is configured to reconstruct the second modality image based on the quantized features to obtain a reconstructed second modality image. The classifier is configured to determine a classification loss and penalize poorly reconstructed regions in the reconstructed image to improve the quality of the reconstructed image. The base model is trained using a preset total loss function, aligning the first image features extracted by the first encoder of the trained model with the second image features extracted by the second encoder, thus obtaining the cross-modal style transfer model; the total loss function is calculated as follows: ,in, These represent the preset weight values ​​for reconstruction loss, quantization loss, similarity loss, and classification loss, respectively. Quantifying loss The calculation formula is expressed as follows: ,in, detach () indicates that gradient backpropagation is paused during model training. This represents the quantization feature; Classification loss The calculation formula is expressed as follows: , Represents a classifier. x c This represents the second modality type diagram; Similarity loss The calculation formula is expressed as follows: ; Reconstruction loss The calculation formula is expressed as follows: ,in, x c This represents the second modality type image. This indicates that the image is reconstructed using the second modality type; The cross-modal style transfer model uses a first encoder to extract target image features from the initial image, and uses a quantization module to quantize the target image features to obtain target quantized features. The decoder decodes and reconstructs the target quantized features to obtain a target image of the second modality.

2. The cross-modal image style transfer method according to claim 1, characterized in that, The method further includes converting the target image features into target fusion features using a pre-constructed feature space diffusion model; the cross-modal style transfer model reconstructs a target image of a second modality based on the target fusion features; The feature space diffusion model is established in advance through the following steps: Collect a diffusion learning sample set, wherein each sample in the diffusion learning sample set includes the first image feature and the second image feature; Design a basic diffusion model, which includes a dynamic similarity mask module and a diffusion module. The dynamic similarity mask module is configured to obtain a mask for its input image features. Using the aforementioned diffusion learning sample set, the basic diffusion model is optimized and trained in the following manner: The first image feature and the second image feature are input into the dynamic similarity mask module to obtain a dynamic similarity mask for the first image feature and the second image feature; The first image feature and the dynamic similarity mask are input into the diffusion module to obtain the first fused feature; The basic diffusion model is trained using a preset diffusion loss function, so that the first fusion feature output by the trained model incorporates the structural information of the second image feature on the basis of the first image feature, thus obtaining the feature space diffusion model; the preset diffusion loss function is at least related to the dynamic similarity mask; The feature space diffusion model utilizes its diffusion module to incorporate the structural information of the image features of the second modality type image into the target image features to obtain the first target fusion feature, and determines the target fusion feature based on the first target fusion feature.

3. The cross-modal image style transfer method according to claim 2, characterized in that, The basic diffusion model further includes a noise processing module, which is configured to add noise and / or denoise the first fusion feature to obtain a second fusion feature; the diffusion loss function is also related to at least the input noise and output noise of the basic diffusion model, wherein the input noise is determined by the first image feature and the second image feature, and the output noise is determined by the second fusion feature; The feature space diffusion model uses the noise processing module therein to add noise and / or denoise the first target fusion feature to obtain the second target fusion feature; And determine the second target fusion feature as the target fusion feature.

4. The cross-modal image style transfer method according to claim 2, characterized in that, The diffusion loss function is expressed by the following formula: ， in, DSM This refers to the dynamic similarity mask. This represents the first image feature. This represents the second image feature. This represents the dot product operation. Indicates each time step t Below, from the standard normal distribution ( Random noise sampled in ) Combined with input data and Perform multiple samplings and output noise With input noise Expected error between Indicates input noise. Indicates output noise. This represents the square of the L2 norm.

5. The cross-modal image style transfer method according to claim 2 or 3, characterized in that, The formula for calculating the dynamic similarity mask is as follows: ； Wherein, DSM represents the dynamic similarity mask. This indicates the current training epoch number, and Total epochs indicates the total number of training epochs. The cosine similarity between the first image feature and the second image feature is represented. The range of values ​​is , Change the range of similarity values ​​to , This means finding the minimum value between two elements. This is the scaling factor, which is a preset value.

6. The cross-modal image style transfer method according to claim 2, characterized in that, Also includes: The first target fusion feature is determined to be the target fusion feature.

7. A cross-modal image style transfer system, characterized in that, The cross-modal image style transfer system uses the cross-modal image style transfer method as described in any one of claims 1 to 6 to convert an image of a first modality type into an image of a second modality type.

8. A computer-readable storage medium for storing program instructions, characterized in that, The program instructions are configured to be invoked to perform the steps of the method as described in any one of claims 1 to 6.

Images